2024-09-15T01:55:23.321 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T01:55:23.334 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T01:55:23.408 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905101 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} email: skanta@redhat.com first_in_suite: false job_id: '7905101' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXEjY4Ra80GEVyWTfHNjDR3XsMm+cueaRc9pdldu1yL5NEZ6/15raHsn+WFocs96gf/vv2G1GQiOEuXhE48LWE= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T01:55:23.408 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T01:55:23.409 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T01:55:23.409 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T01:55:23.410 INFO:teuthology.task.internal:Checking packages... 2024-09-15T01:55:23.432 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T01:55:23.433 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T01:55:23.433 INFO:teuthology.packaging:ref: None 2024-09-15T01:55:23.433 INFO:teuthology.packaging:tag: None 2024-09-15T01:55:23.433 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T01:55:23.433 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T01:55:23.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T01:55:23.574 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T01:55:23.576 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T01:55:23.583 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T01:55:23.583 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T01:55:23.590 INFO:teuthology.task.internal:Saving configuration 2024-09-15T01:55:23.599 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T01:55:23.606 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T01:55:23.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905101', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 01:50:33.121326', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXEjY4Ra80GEVyWTfHNjDR3XsMm+cueaRc9pdldu1yL5NEZ6/15raHsn+WFocs96gf/vv2G1GQiOEuXhE48LWE='} 2024-09-15T01:55:23.626 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T01:55:23.633 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-15T01:55:23.633 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T01:55:23.670 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f494a2f3be0>, signals=[15]) 2024-09-15T01:55:23.670 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T01:55:23.676 INFO:teuthology.task.internal:Opening connections... 2024-09-15T01:55:23.676 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-09-15T01:55:23.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:55:23.755 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T01:55:23.762 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-09-15T01:55:23.834 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-09-15T01:55:23.834 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="9" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="9" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el9" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-09-15T01:55:23.892 INFO:teuthology.orchestra.run.smithi130.stdout:LOGO="fedora-logo-icon" 2024-09-15T01:55:23.893 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T01:55:23.893 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2024-09-15T01:55:23.893 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T01:55:23.893 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T01:55:23.893 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T01:55:23.893 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-09-15T01:55:23.914 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T01:55:23.922 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T01:55:23.943 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T01:55:23.944 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-09-15T01:55:23.960 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T01:55:23.966 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T01:55:23.966 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-09-15T01:55:24.100 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T01:55:24.133 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T01:55:24.148 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T01:55:24.148 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T01:55:24.148 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T01:55:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:55:24.165 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-09-15T01:55:24.224 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-503.el9.x86_64 2024-09-15T01:55:24.224 INFO:teuthology.task.kernel:Running kernel on smithi130: 5.14.0-503.el9.x86_64 2024-09-15T01:55:24.224 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum install -y kernel 2024-09-15T01:55:26.710 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-15T01:55:29.510 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-15T01:55:33.778 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T01:55:35.237 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 19 kB 00:00 2024-09-15T01:55:36.016 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-15T01:55:41.455 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T01:55:43.495 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T01:55:43.495 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T01:55:43.592 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T01:55:43.608 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T01:55:43.609 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T01:55:43.609 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2024-09-15T01:55:43.609 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T01:55:43.609 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 79 M 2024-09-15T01:55:43.610 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 126 M 2024-09-15T01:55:43.610 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T01:55:43.957 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 241 kB/s | 23 kB 00:00 2024-09-15T01:55:46.515 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-09-15T01:55:47.665 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 7.7 MB/s | 29 MB 00:03 2024-09-15T01:55:50.165 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 5.5 MB/s | 35 MB 00:06 2024-09-15T01:55:50.165 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T01:55:50.166 INFO:teuthology.orchestra.run.smithi130.stdout:Total 12 MB/s | 79 MB 00:06 2024-09-15T01:55:50.849 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T01:55:51.225 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T01:55:51.225 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T01:55:52.148 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T01:55:52.149 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T01:55:54.593 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T01:55:55.258 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T01:55:55.350 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:55:58.131 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:55:58.262 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:56:03.101 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:56:03.221 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:56:08.068 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:56:46.588 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:56:46.637 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:56:47.026 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:56:47.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T01:56:47.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T01:56:47.026 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T01:56:47.866 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T01:56:47.867 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T01:56:48.217 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T01:56:48.304 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T01:56:48.305 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T01:56:48.305 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T01:56:48.305 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T01:56:48.305 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T01:56:48.305 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T01:56:48.306 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T01:56:48.306 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T01:56:48.364 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T01:56:48.364 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T01:56:48.364 DEBUG:teuthology.orchestra.run.smithi130:> sudo rpm -qi grub2-tools 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Name : grub2-tools 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Epoch : 1 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Version : 2.06 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Release : 82.el9 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: x86_64 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Group : Unspecified 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:Size : 8274670 2024-09-15T01:56:48.396 INFO:teuthology.orchestra.run.smithi130.stdout:License : GPLv3+ 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Packager : builder@centos.org 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Vendor : CentOS 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Summary : Support tools for GRUB. 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:Description : 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T01:56:48.397 INFO:teuthology.orchestra.run.smithi130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T01:56:48.398 INFO:teuthology.orchestra.run.smithi130.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T01:56:48.398 INFO:teuthology.orchestra.run.smithi130.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T01:56:48.398 INFO:teuthology.orchestra.run.smithi130.stdout:hardware devices. 2024-09-15T01:56:48.398 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T01:56:48.398 INFO:teuthology.orchestra.run.smithi130.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T01:56:48.399 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T01:56:48.399 INFO:teuthology.task.kernel:Updating grub on smithi130 to boot 5.14.0-505.el9.x86_64 2024-09-15T01:56:48.399 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T01:56:49.092 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2024-09-15T01:56:50.425 INFO:teuthology.orchestra.run.smithi130.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T01:56:50.454 INFO:teuthology.orchestra.run.smithi130.stderr:done 2024-09-15T01:56:50.456 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-09-15T01:56:50.471 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.5ZAzkN6Esw 2024-09-15T01:56:50.471 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5ZAzkN6Esw 2024-09-15T01:56:50.538 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.5ZAzkN6Esw 2024-09-15T01:56:50.679 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.5ZAzkN6Esw is 6KB 2024-09-15T01:56:50.731 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.5ZAzkN6Esw 2024-09-15T01:56:50.746 DEBUG:teuthology.orchestra.run.smithi130:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T01:56:50.809 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T01:56:50.809 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T01:56:50.809 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T01:56:50.809 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T01:56:50.811 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T01:56:50.963 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2024-09-15T01:57:20.995 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T01:57:20.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-15T01:57:20.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:58:21.058 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T01:58:30.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-15T01:58:30.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T01:58:30.471 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-09-15T01:58:30.893 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-15T01:58:30.893 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T01:58:30.893 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-09-15T01:58:30.909 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-505.el9.x86_64 2024-09-15T01:58:30.909 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T01:58:30.909 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T01:58:30.909 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T01:58:31.910 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T01:58:31.910 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T01:58:31.994 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2024-09-15T01:58:32.013 DEBUG:teuthology.parallel:result is None 2024-09-15T01:58:32.014 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T01:58:32.021 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T01:58:32.021 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T01:58:32.036 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T01:58:32.042 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T01:58:32.050 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T01:58:32.050 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T01:58:32.093 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T01:58:32.099 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T01:58:32.100 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T01:58:32.154 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T01:58:32.164 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T01:58:32.165 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T01:58:32.173 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T01:58:32.173 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T01:58:32.229 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T01:58:32.239 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T01:58:32.239 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T01:58:32.282 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T01:58:32.365 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T01:58:32.413 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T01:58:32.495 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T01:58:32.534 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-15T01:58:32.534 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T01:58:32.600 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-09-15T01:58:32.685 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T01:58:33.020 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T01:58:33.030 INFO:teuthology.task.internal:Starting timer... 2024-09-15T01:58:33.030 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T01:58:33.043 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T01:58:33.052 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-09-15T01:58:33.157 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-09-15T01:58:33.159 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T01:58:33.159 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-09-15T01:58:33.179 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-09-15T01:58:33.179 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2024-09-15T01:58:33.180 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T01:58:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T01:58:33.245 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T01:58:33.245 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-09-15T01:58:33.338 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T01:58:33.348 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T01:58:33.361 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T01:58:33.361 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T02:02:16.611 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2024-09-15T02:02:16.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-15T02:02:16.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:02:16.692 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-09-15T02:02:16.777 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-15T02:02:16.777 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T02:02:16.790 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T02:02:16.791 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:02:16.791 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:02:16.846 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T02:02:16.860 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T02:02:16.886 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-09-15T02:02:16.897 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-09-15T02:02:16.911 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T02:02:16.925 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T02:02:16.983 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-09-15T02:02:16.987 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T02:02:16.987 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-09-15T02:02:16.987 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:02:16.987 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:02:16.987 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:02:16.987 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T02:02:16.988 INFO:teuthology.run_tasks:Running task install... 2024-09-15T02:02:16.996 DEBUG:teuthology.task.install:project ceph 2024-09-15T02:02:16.996 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T02:02:16.996 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T02:02:16.996 INFO:teuthology.task.install:Using flavor: default 2024-09-15T02:02:17.002 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T02:02:17.002 INFO:teuthology.task.install:extra packages: [] 2024-09-15T02:02:17.002 DEBUG:teuthology.orchestra.run.smithi130:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T02:02:17.586 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:44 ago on Sun 15 Sep 2024 02:01:33 AM UTC. 2024-09-15T02:02:17.674 INFO:teuthology.orchestra.run.smithi130.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T02:02:17.716 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:02:17.717 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:02:17.717 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:02:17.781 DEBUG:teuthology.orchestra.run.smithi130:> sudo dnf -y copr enable ceph/el9 2024-09-15T02:02:17.952 INFO:teuthology.orchestra.run.smithi130.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T02:02:17.952 INFO:teuthology.orchestra.run.smithi130.stderr:of the main distribution, and quality may vary. 2024-09-15T02:02:17.952 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-09-15T02:02:17.952 INFO:teuthology.orchestra.run.smithi130.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T02:02:17.952 INFO:teuthology.orchestra.run.smithi130.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T02:02:17.953 INFO:teuthology.orchestra.run.smithi130.stderr:, 2024-09-15T02:02:17.953 INFO:teuthology.orchestra.run.smithi130.stderr:and packages are not held to any quality or security level. 2024-09-15T02:02:17.953 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-09-15T02:02:17.953 INFO:teuthology.orchestra.run.smithi130.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T02:02:17.953 INFO:teuthology.orchestra.run.smithi130.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T02:02:18.004 INFO:teuthology.orchestra.run.smithi130.stdout:Repository successfully enabled. 2024-09-15T02:02:18.039 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T02:02:18.040 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:02:18.183 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T02:02:18.184 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T02:02:18.294 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T02:02:18.294 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-15T02:02:18.294 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T02:02:18.320 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T02:02:18.320 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T02:02:18.385 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T02:02:18.461 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T02:02:18.525 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-09-15T02:02:18.526 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-09-15T02:02:18.753 INFO:teuthology.orchestra.run.smithi130.stdout:45 files removed 2024-09-15T02:02:18.780 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw 2024-09-15T02:02:19.329 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 292 kB/s | 77 kB 00:00 2024-09-15T02:02:19.596 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-15T02:02:19.864 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-15T02:02:20.023 INFO:teuthology.orchestra.run.smithi130.stdout:Copr repo for el9 owned by ceph 61 kB/s | 7.6 kB 00:00 2024-09-15T02:02:20.456 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 21 MB/s | 8.2 MB 00:00 2024-09-15T02:02:22.191 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-15T02:02:26.383 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-15T02:02:27.801 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-15T02:02:28.563 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-15T02:02:33.986 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-09-15T02:02:36.436 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:02:36.443 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:02:36.444 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T02:02:36.445 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout:Install 22 Packages 2024-09-15T02:02:36.446 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 2 Packages 2024-09-15T02:02:36.447 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:02:36.447 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 58 M 2024-09-15T02:02:36.447 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:02:37.061 INFO:teuthology.orchestra.run.smithi130.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 8.8 MB/s | 5.0 MB 00:00 2024-09-15T02:02:37.137 INFO:teuthology.orchestra.run.smithi130.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 336 kB/s | 25 kB 00:00 2024-09-15T02:02:37.245 INFO:teuthology.orchestra.run.smithi130.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 5.7 MB/s | 637 kB 00:00 2024-09-15T02:02:37.345 INFO:teuthology.orchestra.run.smithi130.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 4.5 MB/s | 458 kB 00:00 2024-09-15T02:02:37.496 INFO:teuthology.orchestra.run.smithi130.stdout:(5/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 14 MB/s | 14 MB 00:00 2024-09-15T02:02:37.546 INFO:teuthology.orchestra.run.smithi130.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 905 kB/s | 45 kB 00:00 2024-09-15T02:02:37.638 INFO:teuthology.orchestra.run.smithi130.stdout:(7/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 17 MB/s | 4.9 MB 00:00 2024-09-15T02:02:37.688 INFO:teuthology.orchestra.run.smithi130.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 847 kB/s | 120 kB 00:00 2024-09-15T02:02:37.722 INFO:teuthology.orchestra.run.smithi130.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 1.8 MB/s | 156 kB 00:00 2024-09-15T02:02:37.755 INFO:teuthology.orchestra.run.smithi130.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 4.7 MB/s | 320 kB 00:00 2024-09-15T02:02:37.781 INFO:teuthology.orchestra.run.smithi130.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-15T02:02:37.806 INFO:teuthology.orchestra.run.smithi130.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-15T02:02:38.023 INFO:teuthology.orchestra.run.smithi130.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 15 MB/s | 22 MB 00:01 2024-09-15T02:02:38.540 INFO:teuthology.orchestra.run.smithi130.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 62 kB/s | 45 kB 00:00 2024-09-15T02:02:38.657 INFO:teuthology.orchestra.run.smithi130.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 183 kB/s | 160 kB 00:00 2024-09-15T02:02:38.965 INFO:teuthology.orchestra.run.smithi130.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 703 kB/s | 662 kB 00:00 2024-09-15T02:02:39.016 INFO:teuthology.orchestra.run.smithi130.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.2 MB/s | 308 kB 00:00 2024-09-15T02:02:39.041 INFO:teuthology.orchestra.run.smithi130.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-15T02:02:39.066 INFO:teuthology.orchestra.run.smithi130.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-15T02:02:39.116 INFO:teuthology.orchestra.run.smithi130.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 32 MB/s | 1.6 MB 00:00 2024-09-15T02:02:39.258 INFO:teuthology.orchestra.run.smithi130.stdout:(21/24): python3-prettytable-0.7.2-27.el9.noarc 70 kB/s | 42 kB 00:00 2024-09-15T02:02:39.342 INFO:teuthology.orchestra.run.smithi130.stdout:(22/24): librados2-17.2.7-1643.gd75f2382.el9.x8 14 MB/s | 3.1 MB 00:00 2024-09-15T02:02:39.375 INFO:teuthology.orchestra.run.smithi130.stdout:(23/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 350 kB/s | 292 kB 00:00 2024-09-15T02:02:39.501 INFO:teuthology.orchestra.run.smithi130.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-09-15T02:02:39.505 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:02:39.505 INFO:teuthology.orchestra.run.smithi130.stdout:Total 19 MB/s | 58 MB 00:03 2024-09-15T02:02:39.570 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:02:39.604 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:02:39.604 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:02:40.286 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:02:40.286 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:02:41.501 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:02:41.841 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T02:02:41.944 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:02:42.020 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:02:42.097 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:02:42.121 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:02:42.210 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:02:42.260 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T02:02:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T02:02:42.738 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T02:02:42.782 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:02:42.845 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T02:02:42.881 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T02:02:42.924 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T02:02:42.962 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:02:43.023 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:02:43.051 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:02:43.117 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T02:02:43.171 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T02:02:43.209 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T02:02:43.242 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T02:02:43.535 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T02:02:43.588 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:02:43.679 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T02:02:43.752 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T02:02:43.782 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T02:02:46.264 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:02:46.295 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:02:46.851 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T02:02:46.881 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:02:46.979 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T02:02:46.980 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T02:02:46.980 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T02:02:46.980 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:02:47.016 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /root 2024-09-15T02:02:57.059 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-09-15T02:02:57.060 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:02:58.412 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:02:58.491 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T02:02:58.492 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:02:58.492 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T02:02:58.492 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:02:58.492 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T02:02:58.492 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:02:58.493 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:02:58.574 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T02:02:58.574 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:02:59.853 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:02:59.853 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T02:02:59.854 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T02:02:59.855 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T02:03:00.771 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T02:03:00.771 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:00.771 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-09-15T02:03:00.771 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.772 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:00.773 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:01.136 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-test 2024-09-15T02:03:01.698 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:01.831 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:01.831 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2024-09-15T02:03:01.832 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:01.833 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 48 M 2024-09-15T02:03:01.833 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 193 M 2024-09-15T02:03:01.833 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:02.094 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-15T02:03:02.178 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 664 kB/s | 163 kB 00:00 2024-09-15T02:03:02.203 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 588 kB/s | 64 kB 00:00 2024-09-15T02:03:04.161 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 21 MB/s | 48 MB 00:02 2024-09-15T02:03:04.163 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:04.164 INFO:teuthology.orchestra.run.smithi130.stdout:Total 21 MB/s | 48 MB 00:02 2024-09-15T02:03:04.191 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:04.211 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:04.211 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:04.577 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:04.577 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:05.098 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:05.191 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T02:03:05.240 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T02:03:05.263 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:03:09.807 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T02:03:09.877 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:03:12.074 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T02:03:12.074 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T02:03:12.074 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T02:03:12.074 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T02:03:12.402 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:12.403 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:12.679 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph 2024-09-15T02:03:13.246 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:40 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:13.382 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:13.384 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:13.384 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:13.384 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:13.384 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:13.384 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T02:03:13.384 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T02:03:13.385 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T02:03:13.386 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout:Install 34 Packages 2024-09-15T02:03:13.387 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:13.388 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 27 M 2024-09-15T02:03:13.388 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 96 M 2024-09-15T02:03:13.388 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:13.910 INFO:teuthology.orchestra.run.smithi130.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2024-09-15T02:03:14.594 INFO:teuthology.orchestra.run.smithi130.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 2.1 MB/s | 2.1 MB 00:00 2024-09-15T02:03:14.636 INFO:teuthology.orchestra.run.smithi130.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 1.4 MB/s | 1.4 MB 00:01 2024-09-15T02:03:14.744 INFO:teuthology.orchestra.run.smithi130.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.2 MB/s | 240 kB 00:00 2024-09-15T02:03:14.869 INFO:teuthology.orchestra.run.smithi130.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-09-15T02:03:14.953 INFO:teuthology.orchestra.run.smithi130.stdout:(6/34): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-15T02:03:14.995 INFO:teuthology.orchestra.run.smithi130.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T02:03:15.037 INFO:teuthology.orchestra.run.smithi130.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-15T02:03:15.079 INFO:teuthology.orchestra.run.smithi130.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-15T02:03:15.121 INFO:teuthology.orchestra.run.smithi130.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 5.1 MB/s | 215 kB 00:00 2024-09-15T02:03:15.229 INFO:teuthology.orchestra.run.smithi130.stdout:(11/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-15T02:03:15.271 INFO:teuthology.orchestra.run.smithi130.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 836 kB/s | 35 kB 00:00 2024-09-15T02:03:15.305 INFO:teuthology.orchestra.run.smithi130.stdout:(13/34): python3-packaging-20.9-5.el9.noarch.rp 2.3 MB/s | 77 kB 00:00 2024-09-15T02:03:15.347 INFO:teuthology.orchestra.run.smithi130.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-15T02:03:15.389 INFO:teuthology.orchestra.run.smithi130.stdout:(15/34): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-15T02:03:15.431 INFO:teuthology.orchestra.run.smithi130.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-15T02:03:15.456 INFO:teuthology.orchestra.run.smithi130.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-09-15T02:03:15.489 INFO:teuthology.orchestra.run.smithi130.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-15T02:03:15.515 INFO:teuthology.orchestra.run.smithi130.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-09-15T02:03:15.540 INFO:teuthology.orchestra.run.smithi130.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-15T02:03:15.565 INFO:teuthology.orchestra.run.smithi130.stdout:(21/34): python3-jaraco-collections-3.0.0-8.el9 927 kB/s | 23 kB 00:00 2024-09-15T02:03:15.591 INFO:teuthology.orchestra.run.smithi130.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 773 kB/s | 19 kB 00:00 2024-09-15T02:03:15.615 INFO:teuthology.orchestra.run.smithi130.stdout:(23/34): python3-jaraco-text-3.2.0-6.el9.noarch 796 kB/s | 20 kB 00:00 2024-09-15T02:03:15.641 INFO:teuthology.orchestra.run.smithi130.stdout:(24/34): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-15T02:03:15.658 INFO:teuthology.orchestra.run.smithi130.stdout:(25/34): python3-more-itertools-8.12.0-2.el9.no 4.6 MB/s | 79 kB 00:00 2024-09-15T02:03:15.683 INFO:teuthology.orchestra.run.smithi130.stdout:(26/34): python3-pecan-1.4.2-3.el9.noarch.rpm 11 MB/s | 272 kB 00:00 2024-09-15T02:03:15.708 INFO:teuthology.orchestra.run.smithi130.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 655 kB/s | 16 kB 00:00 2024-09-15T02:03:15.734 INFO:teuthology.orchestra.run.smithi130.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-15T02:03:15.759 INFO:teuthology.orchestra.run.smithi130.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-15T02:03:15.784 INFO:teuthology.orchestra.run.smithi130.stdout:(30/34): python3-webob-1.8.8-2.el9.noarch.rpm 9.0 MB/s | 230 kB 00:00 2024-09-15T02:03:15.818 INFO:teuthology.orchestra.run.smithi130.stdout:(31/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 13 MB/s | 427 kB 00:00 2024-09-15T02:03:15.843 INFO:teuthology.orchestra.run.smithi130.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-15T02:03:16.176 INFO:teuthology.orchestra.run.smithi130.stdout:(33/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86 1.7 MB/s | 3.9 MB 00:02 2024-09-15T02:03:17.177 INFO:teuthology.orchestra.run.smithi130.stdout:(34/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 5.9 MB/s | 15 MB 00:02 2024-09-15T02:03:17.180 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:17.180 INFO:teuthology.orchestra.run.smithi130.stdout:Total 7.2 MB/s | 27 MB 00:03 2024-09-15T02:03:17.310 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:17.347 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:17.347 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:17.746 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:17.747 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:18.995 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:19.046 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T02:03:19.070 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T02:03:19.115 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T02:03:19.207 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T02:03:19.285 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T02:03:19.325 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T02:03:19.360 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T02:03:19.390 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T02:03:19.425 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T02:03:19.458 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T02:03:19.499 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T02:03:19.534 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T02:03:19.568 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T02:03:19.608 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T02:03:19.640 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:03:19.714 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T02:03:19.870 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T02:03:19.955 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T02:03:20.009 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T02:03:20.047 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T02:03:20.105 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T02:03:20.160 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T02:03:20.348 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T02:03:20.402 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T02:03:20.479 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T02:03:20.568 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T02:03:20.623 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T02:03:20.739 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T02:03:20.887 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T02:03:20.917 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:03:20.943 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T02:03:20.944 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:03:20.944 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T02:03:20.944 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:03:20.944 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T02:03:20.944 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:22.267 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:03:22.295 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T02:03:22.295 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:03:22.295 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T02:03:22.295 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:03:22.295 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T02:03:22.296 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:22.668 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:03:22.692 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T02:03:22.692 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:03:22.692 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T02:03:22.692 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:03:22.692 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T02:03:22.693 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:22.894 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:03:22.917 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T02:03:22.918 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:03:22.918 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T02:03:22.918 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:03:22.918 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T02:03:22.918 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:22.981 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:03:24.740 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T02:03:24.740 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T02:03:24.740 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T02:03:24.740 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T02:03:24.740 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T02:03:24.741 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T02:03:24.742 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T02:03:24.743 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T02:03:24.743 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T02:03:24.743 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T02:03:24.743 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T02:03:24.743 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T02:03:25.619 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T02:03:25.620 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T02:03:25.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T02:03:25.622 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:25.623 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:25.999 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-base 2024-09-15T02:03:26.577 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:26.668 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:03:26.710 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:26.711 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:03:26.711 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:26.779 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install cephadm 2024-09-15T02:03:27.343 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:27.474 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:27.475 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:27.475 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:27.475 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:27.475 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:27.475 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T02:03:27.475 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:27.475 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:27.476 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:27.476 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:03:27.476 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:27.476 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 83 k 2024-09-15T02:03:27.476 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 375 k 2024-09-15T02:03:27.476 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:27.642 INFO:teuthology.orchestra.run.smithi130.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 499 kB/s | 83 kB 00:00 2024-09-15T02:03:27.642 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:27.643 INFO:teuthology.orchestra.run.smithi130.stdout:Total 494 kB/s | 83 kB 00:00 2024-09-15T02:03:27.643 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:27.648 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:27.648 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:27.650 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:27.650 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:27.725 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:28.183 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:03:28.261 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:03:28.641 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:03:28.883 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:03:28.883 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:28.883 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:28.883 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:28.883 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:28.883 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:29.128 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T02:03:29.704 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:29.841 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:29.842 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:29.843 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:03:29.843 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:29.843 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 137 k 2024-09-15T02:03:29.843 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 379 k 2024-09-15T02:03:29.843 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:30.055 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 650 kB/s | 137 kB 00:00 2024-09-15T02:03:30.055 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:30.056 INFO:teuthology.orchestra.run.smithi130.stdout:Total 644 kB/s | 137 kB 00:00 2024-09-15T02:03:30.056 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:30.063 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:30.063 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:30.120 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:30.120 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:30.272 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:30.356 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:03:30.380 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:03:30.380 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:03:30.381 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T02:03:30.381 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:31.302 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T02:03:31.302 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:31.302 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:31.302 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:03:31.302 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:31.303 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:31.528 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr 2024-09-15T02:03:32.106 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:59 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:32.199 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:03:32.242 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:32.243 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:03:32.243 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:32.310 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T02:03:32.868 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:59 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:33.003 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:33.003 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-09-15T02:03:33.004 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T02:03:33.005 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:33.005 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:33.005 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:33.005 INFO:teuthology.orchestra.run.smithi130.stdout:Install 7 Packages 2024-09-15T02:03:33.005 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:33.005 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 3.7 M 2024-09-15T02:03:33.005 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 80 M 2024-09-15T02:03:33.006 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:33.193 INFO:teuthology.orchestra.run.smithi130.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 94 kB/s | 15 kB 00:00 2024-09-15T02:03:33.218 INFO:teuthology.orchestra.run.smithi130.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1643.gd75 135 kB/s | 24 kB 00:00 2024-09-15T02:03:33.260 INFO:teuthology.orchestra.run.smithi130.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T02:03:33.285 INFO:teuthology.orchestra.run.smithi130.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T02:03:33.310 INFO:teuthology.orchestra.run.smithi130.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-15T02:03:33.336 INFO:teuthology.orchestra.run.smithi130.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T02:03:33.402 INFO:teuthology.orchestra.run.smithi130.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 9.2 MB/s | 3.4 MB 00:00 2024-09-15T02:03:33.406 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:33.406 INFO:teuthology.orchestra.run.smithi130.stdout:Total 9.1 MB/s | 3.7 MB 00:00 2024-09-15T02:03:33.407 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:33.415 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:33.415 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:33.502 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:33.502 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:33.803 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:33.862 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T02:03:33.888 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T02:03:33.942 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T02:03:33.974 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T02:03:34.003 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T02:03:35.002 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T02:03:35.032 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:03:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T02:03:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T02:03:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T02:03:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T02:03:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T02:03:36.073 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T02:03:36.074 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T02:03:36.369 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:36.370 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:36.607 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T02:03:37.176 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:04 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:37.310 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:37.311 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-09-15T02:03:37.311 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:37.311 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T02:03:37.312 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout:Install 12 Packages 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 38 M 2024-09-15T02:03:37.313 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 212 M 2024-09-15T02:03:37.314 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:37.668 INFO:teuthology.orchestra.run.smithi130.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-15T02:03:37.768 INFO:teuthology.orchestra.run.smithi130.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 801 kB 00:00 2024-09-15T02:03:37.793 INFO:teuthology.orchestra.run.smithi130.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-15T02:03:37.860 INFO:teuthology.orchestra.run.smithi130.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-09-15T02:03:37.961 INFO:teuthology.orchestra.run.smithi130.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-15T02:03:37.986 INFO:teuthology.orchestra.run.smithi130.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 308 kB/s | 38 kB 00:00 2024-09-15T02:03:38.078 INFO:teuthology.orchestra.run.smithi130.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-09-15T02:03:38.245 INFO:teuthology.orchestra.run.smithi130.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2024-09-15T02:03:38.320 INFO:teuthology.orchestra.run.smithi130.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-09-15T02:03:38.345 INFO:teuthology.orchestra.run.smithi130.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-15T02:03:38.412 INFO:teuthology.orchestra.run.smithi130.stdout:(11/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 4.6 MB/s | 3.0 MB 00:00 2024-09-15T02:03:38.721 INFO:teuthology.orchestra.run.smithi130.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-15T02:03:38.721 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:38.722 INFO:teuthology.orchestra.run.smithi130.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-15T02:03:39.078 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:39.103 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:39.104 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:39.606 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:39.607 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:40.358 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:40.435 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T02:03:40.480 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:03:40.577 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:03:40.612 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T02:03:40.795 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T02:03:40.827 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T02:03:40.913 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T02:03:41.558 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T02:03:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T02:03:42.916 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:03:43.935 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T02:03:43.958 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:03:46.559 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T02:03:46.559 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T02:03:46.559 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T02:03:46.559 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T02:03:46.560 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T02:03:46.991 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T02:03:46.992 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T02:03:46.993 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T02:03:46.993 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:46.993 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:47.380 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-rook 2024-09-15T02:03:47.965 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:14 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:48.113 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T02:03:48.114 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout:Install 13 Packages 2024-09-15T02:03:48.115 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:48.116 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 2.2 M 2024-09-15T02:03:48.116 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 26 M 2024-09-15T02:03:48.116 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:48.641 INFO:teuthology.orchestra.run.smithi130.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 380 kB/s | 26 kB 00:00 2024-09-15T02:03:48.666 INFO:teuthology.orchestra.run.smithi130.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 207 kB/s | 19 kB 00:00 2024-09-15T02:03:48.724 INFO:teuthology.orchestra.run.smithi130.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-09-15T02:03:48.750 INFO:teuthology.orchestra.run.smithi130.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-15T02:03:48.775 INFO:teuthology.orchestra.run.smithi130.stdout:(5/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 235 kB/s | 47 kB 00:00 2024-09-15T02:03:48.800 INFO:teuthology.orchestra.run.smithi130.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-15T02:03:48.834 INFO:teuthology.orchestra.run.smithi130.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-09-15T02:03:48.893 INFO:teuthology.orchestra.run.smithi130.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-09-15T02:03:48.918 INFO:teuthology.orchestra.run.smithi130.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 120 kB/s | 14 kB 00:00 2024-09-15T02:03:48.943 INFO:teuthology.orchestra.run.smithi130.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T02:03:48.968 INFO:teuthology.orchestra.run.smithi130.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-09-15T02:03:48.994 INFO:teuthology.orchestra.run.smithi130.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-15T02:03:49.027 INFO:teuthology.orchestra.run.smithi130.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-15T02:03:49.030 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:49.030 INFO:teuthology.orchestra.run.smithi130.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-15T02:03:49.094 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:49.109 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:49.109 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:49.197 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:49.197 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:49.703 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:49.881 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T02:03:49.937 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T02:03:50.001 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T02:03:50.038 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T02:03:50.079 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T02:03:50.182 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T02:03:50.282 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T02:03:50.339 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T02:03:51.050 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T02:03:51.094 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T02:03:51.128 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T02:03:51.173 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T02:03:51.197 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:03:51.963 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T02:03:51.963 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T02:03:51.964 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T02:03:51.965 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T02:03:52.290 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T02:03:52.290 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:52.290 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:52.290 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:52.290 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:52.291 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:52.565 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T02:03:53.135 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:20 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:53.268 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:53.269 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:53.269 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:53.269 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:53.270 INFO:teuthology.orchestra.run.smithi130.stdout:Install 6 Packages 2024-09-15T02:03:53.271 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:53.271 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 7.0 M 2024-09-15T02:03:53.271 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 33 M 2024-09-15T02:03:53.271 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:53.595 INFO:teuthology.orchestra.run.smithi130.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-15T02:03:53.620 INFO:teuthology.orchestra.run.smithi130.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 578 kB/s | 124 kB 00:00 2024-09-15T02:03:53.670 INFO:teuthology.orchestra.run.smithi130.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-15T02:03:53.704 INFO:teuthology.orchestra.run.smithi130.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 687 kB/s | 58 kB 00:00 2024-09-15T02:03:53.729 INFO:teuthology.orchestra.run.smithi130.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-15T02:03:53.854 INFO:teuthology.orchestra.run.smithi130.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-15T02:03:53.857 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:53.857 INFO:teuthology.orchestra.run.smithi130.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-15T02:03:53.929 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:53.941 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:53.941 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:54.040 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:54.040 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:54.349 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:54.529 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T02:03:54.573 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T02:03:55.112 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T02:03:55.192 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T02:03:55.247 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T02:03:55.285 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:03:56.091 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T02:03:56.092 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T02:03:56.363 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:56.364 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:56.612 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-fuse 2024-09-15T02:03:57.178 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:24 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:03:57.311 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:03:57.312 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:57.312 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:03:57.312 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:57.312 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:03:57.312 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T02:03:57.312 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-15T02:03:57.312 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T02:03:57.313 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:57.313 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:03:57.313 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:03:57.313 INFO:teuthology.orchestra.run.smithi130.stdout:Install 2 Packages 2024-09-15T02:03:57.313 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:57.313 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 893 k 2024-09-15T02:03:57.314 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 2.6 M 2024-09-15T02:03:57.314 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:03:57.547 INFO:teuthology.orchestra.run.smithi130.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 615 kB/s | 80 kB 00:00 2024-09-15T02:03:57.714 INFO:teuthology.orchestra.run.smithi130.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-15T02:03:57.714 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:03:57.715 INFO:teuthology.orchestra.run.smithi130.stdout:Total 2.2 MB/s | 893 kB 00:00 2024-09-15T02:03:57.726 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:03:57.748 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:03:57.748 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:03:57.827 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:03:57.827 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:03:58.017 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:03:58.162 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T02:03:58.237 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:03:58.946 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T02:03:58.946 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T02:03:59.213 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T02:03:59.213 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:59.213 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:03:59.213 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T02:03:59.213 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:03:59.213 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:03:59.490 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-volume 2024-09-15T02:04:00.064 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:27 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:00.200 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:00.201 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:04:00.202 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:00.202 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 270 k 2024-09-15T02:04:00.202 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 1.3 M 2024-09-15T02:04:00.202 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:04:00.410 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.3 MB/s | 270 kB 00:00 2024-09-15T02:04:00.410 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:04:00.411 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.3 MB/s | 270 kB 00:00 2024-09-15T02:04:00.411 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:04:00.419 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:04:00.419 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:04:00.453 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:04:00.453 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:04:00.669 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:04:00.736 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:04:00.752 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:04:00.752 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:04:00.753 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T02:04:00.753 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:01.762 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T02:04:01.762 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:01.762 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:04:01.762 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T02:04:01.762 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:01.762 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:02.052 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados-devel 2024-09-15T02:04:02.629 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:02.768 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:04:02.769 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:02.769 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:04:02.769 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:02.769 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 126 k 2024-09-15T02:04:02.769 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 448 k 2024-09-15T02:04:02.769 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:04:02.972 INFO:teuthology.orchestra.run.smithi130.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 622 kB/s | 126 kB 00:00 2024-09-15T02:04:02.973 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:04:02.973 INFO:teuthology.orchestra.run.smithi130.stdout:Total 618 kB/s | 126 kB 00:00 2024-09-15T02:04:02.973 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:04:02.978 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:04:02.979 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:04:03.041 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:04:03.042 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:04:03.192 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:04:03.294 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:03.696 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:03.971 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:03.971 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:03.971 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:04:03.971 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:04:03.972 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:03.972 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:04.198 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs2 2024-09-15T02:04:04.771 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:04.862 INFO:teuthology.orchestra.run.smithi130.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:04:04.906 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:04.907 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:04:04.907 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:04.981 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs-devel 2024-09-15T02:04:05.541 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:05.673 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:05.673 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 25 k 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 110 k 2024-09-15T02:04:05.674 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:04:05.825 INFO:teuthology.orchestra.run.smithi130.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 166 kB/s | 25 kB 00:00 2024-09-15T02:04:05.826 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:04:05.826 INFO:teuthology.orchestra.run.smithi130.stdout:Total 165 kB/s | 25 kB 00:00 2024-09-15T02:04:05.826 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:04:05.829 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:04:05.829 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:04:05.830 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:04:05.831 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:04:05.912 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:04:06.021 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:06.411 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:06.660 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:06.660 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:06.661 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:04:06.661 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:04:06.661 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:06.661 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:06.847 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados2 2024-09-15T02:04:07.411 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:07.502 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:04:07.545 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:07.546 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:04:07.546 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:07.612 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librbd1 2024-09-15T02:04:08.176 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:08.269 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:04:08.312 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:08.313 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:04:08.313 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:08.387 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rados 2024-09-15T02:04:08.952 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:09.043 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:04:09.085 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:09.087 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:04:09.087 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:09.152 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rgw 2024-09-15T02:04:09.718 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:09.809 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:04:09.853 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:09.854 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:04:09.854 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:09.930 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-cephfs 2024-09-15T02:04:10.495 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:10.585 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:04:10.627 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:10.628 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:04:10.628 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:10.696 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rbd 2024-09-15T02:04:11.258 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:11.347 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T02:04:11.389 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:11.390 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-15T02:04:11.390 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:11.459 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-fuse 2024-09-15T02:04:12.033 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:12.167 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:12.168 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 82 k 2024-09-15T02:04:12.169 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 222 k 2024-09-15T02:04:12.169 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:04:12.361 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 425 kB/s | 82 kB 00:00 2024-09-15T02:04:12.361 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:04:12.362 INFO:teuthology.orchestra.run.smithi130.stdout:Total 422 kB/s | 82 kB 00:00 2024-09-15T02:04:12.362 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:04:12.368 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:04:12.369 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:04:12.432 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:04:12.432 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:04:12.576 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:04:12.675 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:13.058 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:13.327 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:13.327 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:13.327 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:04:13.327 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:04:13.328 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:13.328 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:13.577 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-mirror 2024-09-15T02:04:14.148 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:14.281 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:14.282 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:14.283 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-15T02:04:14.283 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:14.283 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:04:14.283 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T02:04:14.283 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:14.283 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:04:14.284 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:14.284 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:04:14.284 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:14.284 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 2.9 M 2024-09-15T02:04:14.284 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 12 M 2024-09-15T02:04:14.284 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:04:14.630 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.5 MB/s | 2.9 MB 00:00 2024-09-15T02:04:14.631 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:04:14.631 INFO:teuthology.orchestra.run.smithi130.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-09-15T02:04:14.631 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:04:14.640 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:04:14.640 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:04:14.719 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:04:14.719 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:04:15.125 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:04:15.189 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:15.214 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:15.214 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T02:04:15.214 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T02:04:15.214 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:04:15.215 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T02:04:15.215 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:16.325 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:16.325 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:16.325 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:04:16.325 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:04:16.325 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:16.325 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:16.575 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-nbd 2024-09-15T02:04:17.165 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 02:02:33 AM UTC. 2024-09-15T02:04:17.300 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-15T02:04:17.301 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:17.301 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-09-15T02:04:17.301 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:17.301 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-15T02:04:17.301 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T02:04:17.301 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:17.301 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-15T02:04:17.302 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-15T02:04:17.302 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-15T02:04:17.302 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:17.302 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 165 k 2024-09-15T02:04:17.302 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 477 k 2024-09-15T02:04:17.302 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-15T02:04:17.516 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 773 kB/s | 165 kB 00:00 2024-09-15T02:04:17.517 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-15T02:04:17.517 INFO:teuthology.orchestra.run.smithi130.stdout:Total 767 kB/s | 165 kB 00:00 2024-09-15T02:04:17.517 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-15T02:04:17.524 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-15T02:04:17.524 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-15T02:04:17.587 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-15T02:04:17.587 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-15T02:04:17.743 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-15T02:04:17.847 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:18.325 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:18.607 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T02:04:18.607 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:18.607 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-15T02:04:18.607 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T02:04:18.607 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-15T02:04:18.607 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-15T02:04:18.948 DEBUG:teuthology.parallel:result is None 2024-09-15T02:04:18.949 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:04:19.112 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T02:04:19.133 INFO:teuthology.orchestra.run.smithi130.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T02:04:19.134 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T02:04:19.134 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T02:04:19.135 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T02:04:19.136 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-15T02:04:19.136 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T02:04:19.199 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T02:04:19.199 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-15T02:04:19.199 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T02:04:19.264 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T02:04:19.332 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T02:04:19.332 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-15T02:04:19.332 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T02:04:19.396 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T02:04:19.462 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T02:04:19.462 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-15T02:04:19.463 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T02:04:19.527 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T02:04:19.593 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T02:04:19.608 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:04:19.609 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T02:04:19.609 INFO:tasks.workunit:timeout=3h 2024-09-15T02:04:19.609 INFO:tasks.workunit:cleanup=True 2024-09-15T02:04:19.609 DEBUG:teuthology.orchestra.run.smithi130:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:04:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:04:19.649 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T02:04:19.650 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T02:04:19.706 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T02:04:19.706 DEBUG:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T02:04:19.761 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:04:19.816 INFO:tasks.workunit.client.0.smithi130.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T02:05:09.437 INFO:tasks.workunit.client.0.smithi130.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-15T02:05:09.437 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-15T02:05:09.437 INFO:tasks.workunit.client.0.smithi130.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T02:05:09.437 INFO:tasks.workunit.client.0.smithi130.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: git switch -c 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:Or undo this operation with: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: git switch - 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-15T02:05:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:05:09.441 DEBUG:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T02:05:09.498 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-15T02:05:09.498 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T02:05:09.553 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-09-15T02:05:09.554 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-09-15T02:05:09.554 DEBUG:teuthology.orchestra.run.smithi130:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-09-15T02:05:09.626 INFO:tasks.workunit.client.0.smithi130.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ TIMEOUT=300 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ MAX_TIMEOUT=15 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ PG_NUM=4 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ TMPDIR=/tmp 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ type xmlstarlet 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:++ XMLSTARLET=xmlstarlet 2024-09-15T02:05:09.627 INFO:tasks.workunit.client.0.smithi130.stderr:+++ uname 2024-09-15T02:05:09.628 INFO:tasks.workunit.client.0.smithi130.stderr:++ '[' Linux = FreeBSD ']' 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:++ SED=sed 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:++ AWK=awk 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:+++ stty -a 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:+++ head -1 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:++ termwidth= 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:++ DIFFCOLOPTS='-y ' 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:++ KERNCORE=kernel.core_pattern 2024-09-15T02:05:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:++ EXTRA_OPTS= 2024-09-15T02:05:09.631 INFO:tasks.workunit.client.0.smithi130.stderr:++ test '' = TESTS 2024-09-15T02:05:09.632 INFO:tasks.workunit.client.0.smithi130.stderr:++ uname -m 2024-09-15T02:05:09.634 INFO:tasks.workunit.client.0.smithi130.stderr:+ arch=x86_64 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ case $arch in 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ plugins=(jerasure shec lrc isa) 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ main test-erasure-code-plugins 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ local dir=td/test-erasure-code-plugins 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ shift 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ shopt -s -o xtrace 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:05:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-erasure-code-plugins 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-09-15T02:05:09.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-09-15T02:05:09.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none ' 2024-09-15T02:05:09.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-09-15T02:05:09.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-09-15T02:05:09.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T02:05:09.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_preload_no_warning 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_preload_no_warning_default 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_preload_warning' 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:05:09.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:05:09.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:05:09.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:05:09.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:05:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:05:09.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:05:09.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:09.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:09.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:05:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:05:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:05:09.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:05:09.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:05:09.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:09.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:09.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:05:09.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:05:09.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:05:10.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:05:10.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:05:10.319 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:10.319 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:10.319 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:05:10.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:05:10.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:05:10.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:05:10.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:05:10.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:05:10.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:05:10.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:05:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:05:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:05:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:05:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:05:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:05:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:05:11.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:05:11.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:05:11.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:05:11.311 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:05:11.311 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:11.311 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:11.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:05:11.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:05:11.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:05:11.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-09-15T02:05:11.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-15T02:05:11.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-09-15T02:05:11.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:11.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:05:11.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:05:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:05:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6648bc57-feb7-49ae-9d79-1aaf92e8c520 2024-09-15T02:05:11.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6648bc57-feb7-49ae-9d79-1aaf92e8c520' 2024-09-15T02:05:11.351 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 6648bc57-feb7-49ae-9d79-1aaf92e8c520 2024-09-15T02:05:11.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:05:11.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXQOZmfGZfFRAAvtlhRQ3VWrb9HHvq2ll1oQ== 2024-09-15T02:05:11.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXQOZmfGZfFRAAvtlhRQ3VWrb9HHvq2ll1oQ=="}' 2024-09-15T02:05:11.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6648bc57-feb7-49ae-9d79-1aaf92e8c520 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:05:11.663 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:05:11.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:05:11.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXQOZmfGZfFRAAvtlhRQ3VWrb9HHvq2ll1oQ== --osd-uuid 6648bc57-feb7-49ae-9d79-1aaf92e8c520 2024-09-15T02:05:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:11.743+0000 7f58e7b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:11.745+0000 7f58e7b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:11.748 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:11.746+0000 7f58e7b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:11.748 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:11.747+0000 7f58e7b1f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:05:11.749 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:11.747+0000 7f58e7b1f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:05:11.749 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:11.747+0000 7f58e7b1f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:05:11.792 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:11.790+0000 7f58e7b1f440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:05:17.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:05:17.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:05:17.881 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:05:17.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:05:17.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:05:18.391 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:05:18.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:05:18.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:05:18.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:05:18.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:05:18.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:05:18.414 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:18.412+0000 7f3ea1c75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:18.430 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:18.429+0000 7f3ea1c75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:18.431 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:18.430+0000 7f3ea1c75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:05:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:05:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:05:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:05:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:05:18.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:18.775 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:05:18.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:05:18.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:18.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:18.971 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:18.969+0000 7f3ea1c75440 -1 Falling back to public interface 2024-09-15T02:05:19.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:20.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:20.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:20.130 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:05:20.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:05:20.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:20.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:20.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:21.443 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:21.441+0000 7f3ea1c75440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:05:21.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:21.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:21.486 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:05:21.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:05:21.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:21.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:22.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:23.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:23.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:23.005 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:05:23.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:05:23.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:23.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:23.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:24.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:24.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:24.433 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:05:24.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:05:24.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:24.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:24.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:25.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:25.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:25.822 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:05:25.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:05:25.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:25.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:26.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:27.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:27.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:27.270 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:05:27.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:05:27.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:27.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:27.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:28.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:28.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:28.628 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:05:28.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:05:28.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:28.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:29.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:30.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:30.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:30.150 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:05:30.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:05:30.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:30.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:31.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:31.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:31.512 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:05:31.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:05:31.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:31.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:31.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:32.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:32.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:32.892 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:05:32.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:05:32.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:32.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:33.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:34.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:34.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:34.655 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:05:34.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:05:34.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:34.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:35.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:36.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:36.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:36.144 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:05:36.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:05:36.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:36.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:37.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:38.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:38.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:38.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:05:38.141 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:05:38.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:38.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:38.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:39.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:39.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:39.575 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:05:39.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:05:39.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:39.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:40.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:41.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:41.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:41.243 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:05:41.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:05:41.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:41.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:42.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:43.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:43.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:43.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:05:43.754 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:05:43.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:43.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:44.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:45.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:45.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:45.835 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:05:45.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:05:45.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:45.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:05:46.220 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/28085,v1:127.0.0.1:6803/28085] [v2:127.0.0.1:6804/28085,v1:127.0.0.1:6805/28085] exists,up 6648bc57-feb7-49ae-9d79-1aaf92e8c520 2024-09-15T02:05:46.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:05:46.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:05:46.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:05:46.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-15T02:05:46.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-09-15T02:05:46.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:05:46.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:05:46.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f35b8d5-1f7b-4296-894c-b8be5483f56c 2024-09-15T02:05:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7f35b8d5-1f7b-4296-894c-b8be5483f56c' 2024-09-15T02:05:46.228 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 7f35b8d5-1f7b-4296-894c-b8be5483f56c 2024-09-15T02:05:46.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:05:46.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6QOZm998FDhAAglVA8IhoJS3q2va2wE9ZGg== 2024-09-15T02:05:46.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6QOZm998FDhAAglVA8IhoJS3q2va2wE9ZGg=="}' 2024-09-15T02:05:46.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f35b8d5-1f7b-4296-894c-b8be5483f56c -i td/test-erasure-code-plugins/1/new.json 2024-09-15T02:05:46.704 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:05:46.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-09-15T02:05:46.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6QOZm998FDhAAglVA8IhoJS3q2va2wE9ZGg== --osd-uuid 7f35b8d5-1f7b-4296-894c-b8be5483f56c 2024-09-15T02:05:46.737 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:46.736+0000 7f2ecd12c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:46.739 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:46.737+0000 7f2ecd12c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:46.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:46.738+0000 7f2ecd12c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:46.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:46.739+0000 7f2ecd12c440 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-15T02:05:46.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:46.739+0000 7f2ecd12c440 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-15T02:05:46.741 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:46.739+0000 7f2ecd12c440 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-15T02:05:46.786 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:46.784+0000 7f2ecd12c440 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-09-15T02:05:52.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-09-15T02:05:52.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:05:52.691 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:05:52.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:05:52.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:05:53.166 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:05:53.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:05:53.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:05:53.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:05:53.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:05:53.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:05:53.187 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:53.185+0000 7fb7d772c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:53.189 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:53.187+0000 7fb7d772c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:53.190 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:53.188+0000 7fb7d772c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:05:53.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:05:53.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:05:53.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:05:53.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:05:53.553 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:05:53.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:05:53.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:53.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:05:53.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:53.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:05:53.707 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:53.706+0000 7fb7d772c440 -1 Falling back to public interface 2024-09-15T02:05:53.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:54.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:54.908 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:05:54.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:05:54.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:54.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:05:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:56.205 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:05:56.202+0000 7fb7d772c440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:05:56.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:56.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:56.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:05:56.260 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:05:56.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:56.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:05:56.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:57.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:57.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:57.858 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:05:57.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:05:57.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:57.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:05:58.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:05:59.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:05:59.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:05:59.354 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:05:59.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:05:59.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:05:59.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:05:59.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:00.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:00.777 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:06:00.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:00.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:06:00.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:00.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:01.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:02.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:02.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:02.241 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:06:02.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:06:02.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:02.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:02.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:03.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:03.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:03.606 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:06:03.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:06:03.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:03.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:04.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:05.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:05.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:05.013 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:06:05.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:06:05.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:05.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:05.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:06.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:06.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:06.400 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:06:06.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:06:06.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:06.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:06.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:07.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:07.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:07.765 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:06:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:06:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:07.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:08.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:09.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:09.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:09.530 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:06:09.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:06:09.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:09.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:11.348 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:06:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:06:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:11.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:12.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:13.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:13.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:13.085 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:06:13.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:06:13.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:13.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:13.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:14.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:14.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:14.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:06:14.711 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:06:14.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:14.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:15.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:16.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:16.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:16.797 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:06:16.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:06:16.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:16.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:17.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:18.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:18.645 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:06:18.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:18.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:06:18.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:18.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:06:19.057 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/29880,v1:127.0.0.1:6811/29880] [v2:127.0.0.1:6812/29880,v1:127.0.0.1:6813/29880] exists,up 7f35b8d5-1f7b-4296-894c-b8be5483f56c 2024-09-15T02:06:19.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:06:19.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:06:19.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:06:19.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-15T02:06:19.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-09-15T02:06:19.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:06:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:06:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:06:19.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f98dca6-5b48-4f02-8b76-3dcb90999b96 2024-09-15T02:06:19.065 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 7f98dca6-5b48-4f02-8b76-3dcb90999b96 2024-09-15T02:06:19.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7f98dca6-5b48-4f02-8b76-3dcb90999b96' 2024-09-15T02:06:19.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:06:19.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbQeZm9ulYBBAAw4uQMSh4uEH4jLjUb/80ew== 2024-09-15T02:06:19.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbQeZm9ulYBBAAw4uQMSh4uEH4jLjUb/80ew=="}' 2024-09-15T02:06:19.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f98dca6-5b48-4f02-8b76-3dcb90999b96 -i td/test-erasure-code-plugins/2/new.json 2024-09-15T02:06:19.605 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:06:19.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-09-15T02:06:19.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbQeZm9ulYBBAAw4uQMSh4uEH4jLjUb/80ew== --osd-uuid 7f98dca6-5b48-4f02-8b76-3dcb90999b96 2024-09-15T02:06:19.639 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:19.637+0000 7fc72f80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:06:19.640 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:19.639+0000 7fc72f80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:06:19.641 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:19.640+0000 7fc72f80e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:06:19.642 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:19.640+0000 7fc72f80e440 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-15T02:06:19.642 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:19.640+0000 7fc72f80e440 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-15T02:06:19.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:19.640+0000 7fc72f80e440 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-15T02:06:19.720 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:19.719+0000 7fc72f80e440 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-09-15T02:06:25.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-09-15T02:06:25.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:06:25.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:06:25.586 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:06:25.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:06:26.064 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:06:26.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:06:26.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:06:26.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:06:26.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:06:26.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:06:26.085 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:26.083+0000 7f54b762f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:06:26.086 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:26.085+0000 7f54b762f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:06:26.087 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:26.086+0000 7f54b762f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:06:26.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:06:26.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:06:26.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:06:26.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:06:26.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:06:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:26.412 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:06:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:06:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:26.618 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:26.617+0000 7f54b762f440 -1 Falling back to public interface 2024-09-15T02:06:26.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:27.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:27.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:27.762 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:06:27.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:06:27.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:27.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:28.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:29.089 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:06:29.087+0000 7f54b762f440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:06:29.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:29.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:29.119 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:06:29.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:06:29.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:29.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:29.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:30.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:30.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:30.609 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:06:30.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:06:30.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:30.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:30.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:31.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:31.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:31.980 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:06:31.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:06:31.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:31.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:32.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:33.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:33.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:33.372 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:06:33.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:06:33.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:33.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:33.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:34.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:34.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:34.778 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:06:34.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:06:34.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:34.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:35.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:36.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:36.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:36.192 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:06:36.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:06:36.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:36.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:36.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:37.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:37.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:37.632 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:06:37.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:06:37.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:37.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:38.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:39.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:39.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:39.048 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:06:39.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:06:39.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:39.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:39.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:40.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:40.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:40.422 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:06:40.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:06:40.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:40.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:41.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:42.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:42.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:42.100 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:06:42.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:06:42.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:42.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:42.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:43.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:43.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:43.716 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:06:43.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:06:43.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:43.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:44.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:45.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:45.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:45.363 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:06:45.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:06:45.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:45.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:46.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:47.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:47.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:47.132 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:06:47.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:06:47.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:47.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:47.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:49.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:49.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:49.000 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:06:49.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:06:49.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:49.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:49.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:50.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:50.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:50.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:06:50.709 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:06:50.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:50.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:51.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:06:52.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:06:52.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:06:52.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:06:52.560 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:06:52.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:06:52.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:06:52.916 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/31629,v1:127.0.0.1:6819/31629] [v2:127.0.0.1:6820/31629,v1:127.0.0.1:6821/31629] exists,up 7f98dca6-5b48-4f02-8b76-3dcb90999b96 2024-09-15T02:06:52.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:06:52.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:06:52.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:06:52.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-09-15T02:06:52.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T02:06:53.254 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-09-15T02:06:53.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T02:06:53.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:06:53.737 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:06:53.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:06:54.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T02:06:56.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-09-15T02:06:56.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:06:56.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:06:56.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:06:56.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:06:56.650 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:06:56.651 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:06:56.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:06:56.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:06:56.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:06:56.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:06:56.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:06:56.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:06:56.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:06:56.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:06:56.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:06:57.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:06:57.177 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:06:57.177 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:06:57.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:06:57.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:06:57.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:06:57.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-15T02:06:57.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-15T02:06:57.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-15T02:06:57.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:06:57.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:06:57.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672974 2024-09-15T02:06:57.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672974 2024-09-15T02:06:57.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-09-15T02:06:57.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:06:57.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:06:57.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509446 2024-09-15T02:06:57.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509446 2024-09-15T02:06:57.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509446' 2024-09-15T02:06:57.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:06:57.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-15T02:06:57.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:06:57.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:06:57.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-15T02:06:57.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:06:57.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-15T02:06:57.557 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836500 2024-09-15T02:06:57.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-15T02:06:57.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:06:57.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-15T02:06:57.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:06:58.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:06:58.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:06:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-15T02:06:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:06:59.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672974 2024-09-15T02:06:59.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:06:59.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:06:59.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672974 2024-09-15T02:06:59.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:06:59.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672974 2024-09-15T02:06:59.249 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672974 2024-09-15T02:06:59.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-15T02:06:59.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:06:59.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-15T02:06:59.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:06:59.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509446 2024-09-15T02:06:59.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:06:59.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:06:59.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509446 2024-09-15T02:06:59.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:06:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509446 2024-09-15T02:06:59.604 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509446 2024-09-15T02:06:59.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-15T02:06:59.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:06:59.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-15T02:06:59.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:06:59.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:06:59.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:07:00.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:07:00.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T02:07:00.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:07:00.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:07:00.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:07:01.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T02:07:01.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:07:01.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:07:01.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-15T02:07:01.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T02:07:01.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:01.913 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:01.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:07:01.925 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:01.500+0000 7ff50a5f1640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:01.925 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:01.831+0000 7ff50a5f1640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:01.925 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:01.831+0000 7ff50a5f1640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-15T02:07:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T02:07:02.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:02.610 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:07:02.622 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:02.174+0000 7ff50a5f1640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:02.622 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:02.529+0000 7ff50a5f1640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:02.622 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:02.529+0000 7ff50a5f1640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:02.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-15T02:07:02.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-09-15T02:07:03.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T02:07:03.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:03.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:03.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T02:07:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:03.183 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:03.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:07:03.195 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:02.871+0000 7ff50a5f1640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:03.195 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:03.103+0000 7ff50a5f1640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:03.195 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:03.103+0000 7ff50a5f1640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-15T02:07:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-15T02:07:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-09-15T02:07:03.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T02:07:03.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:03.886 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:03.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:07:03.898 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:03.479+0000 7ff50a5f1640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-15T02:07:03.898 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:03.805+0000 7ff50a5f1640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-15T02:07:03.898 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:03.805+0000 7ff50a5f1640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-15T02:07:03.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-15T02:07:03.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-09-15T02:07:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T02:07:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:04.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T02:07:04.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:04.634 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:04.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:07:04.647 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:04.190+0000 7ff50a5f1640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-15T02:07:04.647 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:04.554+0000 7ff50a5f1640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-15T02:07:04.647 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:04.554+0000 7ff50a5f1640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-15T02:07:04.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-15T02:07:04.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:05.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-15T02:07:05.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:05.237 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:05.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:07:05.250 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:04.899+0000 7ff50a5f1640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-15T02:07:05.250 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:05.158+0000 7ff50a5f1640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-15T02:07:05.250 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:07:05.158+0000 7ff50a5f1640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-15T02:07:05.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:07:05.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:05.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:07:05.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:07:05.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:07:05.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:07:05.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:07:05.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:07:05.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:07:05.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:07:05.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:07:05.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:07:05.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:07:05.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:07:05.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:07:05.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:07:05.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:05.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:07:05.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:07:05.362 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:05.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:07:05.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:07:05.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:07:05.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:07:05.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:05.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:05.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:07:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:07:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:07:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T02:07:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-09-15T02:07:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T02:07:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:07:05.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:07:05.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:07:05.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:07:05.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:07:05.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:05.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:07:05.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:07:05.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:07:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:07:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:05.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:07:05.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:07:05.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:07:05.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:07:05.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:07:05.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:05.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:05.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:07:05.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:07:05.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:07:05.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:07:05.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:07:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-09-15T02:07:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:07:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:07:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:07:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:07:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-09-15T02:07:05.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:07:05.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:07:05.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:07:05.733 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:07:05.733 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:05.733 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:05.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:05.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-09-15T02:07:05.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:07:05.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:07:05.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:07:05.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:07:06.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:07:06.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:07:06.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:07:06.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:07:06.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:07:06.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:07:06.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:07:06.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:06.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:07:06.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:07:06.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:07:06.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:07:06.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:07:06.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:07:06.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:07:06.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T02:07:06.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:06.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:06.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:06.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:06.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:06.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:06.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:07:06.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:07.045 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:07:07.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:07.057 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:07:07.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-09-15T02:07:07.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:07:07.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:07:07.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=647b6799-f1ac-4178-b079-7ab44811b980 2024-09-15T02:07:07.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 647b6799-f1ac-4178-b079-7ab44811b980' 2024-09-15T02:07:07.059 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 647b6799-f1ac-4178-b079-7ab44811b980 2024-09-15T02:07:07.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:07:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLQeZmljsJBBAAKbklaZe5rcKKUc6+JXZClw== 2024-09-15T02:07:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLQeZmljsJBBAAKbklaZe5rcKKUc6+JXZClw=="}' 2024-09-15T02:07:07.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 647b6799-f1ac-4178-b079-7ab44811b980 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:07:07.331 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:07:07.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:07:07.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQBLQeZmljsJBBAAKbklaZe5rcKKUc6+JXZClw== --osd-uuid 647b6799-f1ac-4178-b079-7ab44811b980 2024-09-15T02:07:07.364 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:07.363+0000 7f1407b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:07.369 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:07.367+0000 7f1407b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:07.370 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:07.368+0000 7f1407b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:07.370 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:07.369+0000 7f1407b23440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:07:07.370 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:07.369+0000 7f1407b23440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:07:07.370 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:07.369+0000 7f1407b23440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:07:07.409 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:07.408+0000 7f1407b23440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:07:12.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:07:12.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:07:12.759 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:07:12.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:07:12.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:07:13.242 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:07:13.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:07:13.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-09-15T02:07:13.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:07:13.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:07:13.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:07:13.264 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:13.262+0000 7f4f58c14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:13.266 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:13.264+0000 7f4f58c14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:13.267 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:13.265+0000 7f4f58c14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:13.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:07:13.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:07:13.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:07:13.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:07:13.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:07:13.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:13.588 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:07:13.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:07:13.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:13.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:13.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:13.793+0000 7f4f58c14440 -1 Falling back to public interface 2024-09-15T02:07:13.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:14.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:14.941 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:07:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:07:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:15.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:16.262 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:16.260+0000 7f4f58c14440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:07:16.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:16.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:16.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:07:16.297 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:07:16.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:16.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:16.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:17.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:17.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:17.670 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:07:17.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:07:17.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:17.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:18.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:19.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:19.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:19.060 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:07:19.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:07:19.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:19.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:19.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:20.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:20.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:20.416 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:07:20.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:07:20.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:20.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:21.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:21.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:21.776 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:07:21.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:07:21.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:21.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:22.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:23.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:23.294 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:07:23.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:07:23.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:23.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:23.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:24.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:24.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:24.654 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:07:24.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:07:24.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:24.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:25.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:26.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:26.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:26.104 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:07:26.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:07:26.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:26.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:26.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:27.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:27.502 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:07:27.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:27.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:07:27.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:27.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:28.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:29.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:29.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:29.007 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:07:29.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:07:29.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:29.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:29.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:30.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:30.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:30.768 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:07:30.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:07:30.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:30.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:31.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:32.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:32.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:32.495 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:07:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:07:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:33.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:34.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:34.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:34.373 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:07:34.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:07:34.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:34.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:35.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:36.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:36.135 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:07:36.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:36.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:07:36.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:36.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:36.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:37.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:37.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:37.834 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:07:37.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:07:37.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:37.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:38.361 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34221,v1:127.0.0.1:6803/34221] [v2:127.0.0.1:6804/34221,v1:127.0.0.1:6805/34221] exists,up 647b6799-f1ac-4178-b079-7ab44811b980 2024-09-15T02:07:38.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:07:38.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:07:38.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:07:38.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T02:07:38.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:07:38.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:07:38.363 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:38.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:07:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:07:38.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:07:38.438 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:38.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:07:38.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:07:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:07:38.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:07:38.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:07:38.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:07:38.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:07:38.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:07:38.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:07:38.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:07:38.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:07:38.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:38.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:07:38.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:07:38.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:38.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:07:38.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:07:38.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:07:38.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:07:38.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:38.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:38.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:07:38.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:07:38.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:07:38.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:07:38.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:07:38.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:07:38.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:07:38.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:07:38.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:07:38.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:07:38.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:07:38.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:07:38.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:07:38.571 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:07:38.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:38.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:07:38.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:07:38.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:07:38.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:07:38.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:07:38.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:07:38.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:07:38.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:38.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:38.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:07:38.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:07:38.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:07:38.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:07:38.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:07:38.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:38.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:38.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:07:38.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-09-15T02:07:38.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:07:38.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:07:38.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:07:38.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:07:38.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:38.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:38.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:38.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-09-15T02:07:39.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:07:39.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:07:39.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:07:39.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:07:39.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:07:39.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:07:39.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:07:39.154 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:39.154 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:39.154 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:39.154 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:39.154 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:39.154 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:39.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:07:39.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:07:39.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:07:39.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:07:39.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:07:39.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:07:39.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:39.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:07:39.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:07:39.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:07:39.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:07:39.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:07:40.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:07:40.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:07:40.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:07:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:07:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:40.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:40.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:07:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:07:40.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:07:40.238 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:07:40.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:07:40.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-09-15T02:07:40.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:07:40.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:07:40.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7d135f7-424d-4b9a-a2be-b63a9e5ac53c 2024-09-15T02:07:40.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b7d135f7-424d-4b9a-a2be-b63a9e5ac53c' 2024-09-15T02:07:40.252 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 b7d135f7-424d-4b9a-a2be-b63a9e5ac53c 2024-09-15T02:07:40.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:07:40.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsQeZmE1SYDxAAJBQcmGq+llw2S1UQhrfY9Q== 2024-09-15T02:07:40.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsQeZmE1SYDxAAJBQcmGq+llw2S1UQhrfY9Q=="}' 2024-09-15T02:07:40.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7d135f7-424d-4b9a-a2be-b63a9e5ac53c -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:07:40.523 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:07:40.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:07:40.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQBsQeZmE1SYDxAAJBQcmGq+llw2S1UQhrfY9Q== --osd-uuid b7d135f7-424d-4b9a-a2be-b63a9e5ac53c 2024-09-15T02:07:40.557 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:40.555+0000 7f1a49d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:40.570 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:40.568+0000 7f1a49d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:40.571 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:40.569+0000 7f1a49d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:40.571 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:40.570+0000 7f1a49d23440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:07:40.571 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:40.570+0000 7f1a49d23440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:07:40.571 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:40.570+0000 7f1a49d23440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:07:40.609 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:40.608+0000 7f1a49d23440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:07:46.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:07:46.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:07:46.216 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:07:46.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:07:46.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:07:46.693 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:07:46.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:07:46.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-09-15T02:07:46.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:07:46.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:07:46.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:07:46.715 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:46.713+0000 7f3ca3b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:46.716 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:46.715+0000 7f3ca3b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:46.717 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:46.716+0000 7f3ca3b23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:07:47.042 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:07:47.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:47.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:47.234 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:47.232+0000 7f3ca3b23440 -1 Falling back to public interface 2024-09-15T02:07:47.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:48.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:48.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:48.397 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:07:48.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:07:48.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:48.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:48.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:49.719 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:07:49.717+0000 7f3ca3b23440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:07:49.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:49.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:49.749 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:07:49.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:07:49.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:49.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:50.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:51.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:51.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:51.103 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:07:51.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:07:51.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:51.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:51.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:52.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:52.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:52.509 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:07:52.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:07:52.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:52.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:53.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:53.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:53.906 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:07:53.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:07:53.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:53.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:54.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:55.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:55.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:55.375 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:07:55.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:07:55.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:55.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:55.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:56.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:56.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:56.736 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:07:56.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:07:56.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:56.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:57.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:58.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:58.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:58.099 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:07:58.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:07:58.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:58.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:58.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:07:59.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:07:59.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:07:59.516 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:07:59.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:07:59.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:07:59.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:07:59.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:00.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:00.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:00.876 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:08:00.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:08:00.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:00.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:01.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:02.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:02.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:02.235 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:08:02.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:08:02.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:02.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:02.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:03.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:03.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:03.762 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:08:03.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:08:03.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:03.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:04.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:05.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:05.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:05.806 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:08:05.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:08:05.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:05.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:06.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:07.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:07.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:07.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:08:07.614 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:08:07.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:07.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:08.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:09.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:09.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:09.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:08:09.363 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:08:09.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:09.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:10.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:11.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:11.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:11.085 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:08:11.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:08:11.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:11.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:11.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:12.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:12.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:12.441 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:08:12.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:08:12.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:12.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:12.805 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/36225,v1:127.0.0.1:6803/36225] [v2:127.0.0.1:6804/36225,v1:127.0.0.1:6805/36225] exists,up b7d135f7-424d-4b9a-a2be-b63a9e5ac53c 2024-09-15T02:08:12.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:08:12.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:08:12.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:08:12.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T02:08:12.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:08:12.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:08:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:12.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:08:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:08:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:08:12.881 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:08:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:08:12.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:08:12.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:08:12.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:12.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:08:12.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:08:12.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:08:12.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:08:12.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:08:12.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:08:12.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:08:12.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:08:12.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:08:13.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:08:13.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:08:13.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:08:13.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:08:13.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:08:13.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:13.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:08:13.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:08:13.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:13.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:08:13.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:08:13.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:08:13.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:08:13.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:13.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:13.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:08:13.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:08:13.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:08:13.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:08:13.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:08:13.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:08:13.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:08:13.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:08:13.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:08:13.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:08:13.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:08:13.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:08:13.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:08:13.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:08:13.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:13.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:08:13.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:08:13.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:13.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:08:13.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:08:13.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:08:13.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:08:13.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:13.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:13.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:08:13.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:08:13.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:08:13.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:08:13.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:08:13.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:13.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:13.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:08:13.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:08:13.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:08:13.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:08:13.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:08:13.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-09-15T02:08:13.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:13.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:08:13.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:08:13.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:08:13.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:08:13.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-09-15T02:08:13.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:08:13.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:08:13.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:08:13.386 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:08:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:13.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-09-15T02:08:13.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:08:13.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:08:13.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:08:13.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:08:13.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:08:13.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:08:13.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:08:13.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:08:13.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:08:13.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:08:13.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:08:13.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:13.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:08:13.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:08:13.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:08:13.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:08:13.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:08:14.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:08:14.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:08:14.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:08:14.658 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:08:14.658 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:14.658 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:14.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:14.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:08:14.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:08:14.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T02:08:14.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:08:14.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:08:14.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:14.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:14.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:14.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:08:14.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:08:14.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:08:14.750 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:08:14.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-09-15T02:08:14.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:14.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:08:14.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:08:14.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:08:14.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:08:14.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:08:14.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:08:14.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:08:14.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:08:14.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:08:14.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:08:14.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:08:14.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:08:14.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:08:14.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:08:14.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:08:14.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:08:14.767 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:08:14.767 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:14.767 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:14.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:14.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:14.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:08:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:08:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:08:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:08:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:08:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:08:14.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:08:14.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:08:14.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:08:14.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:08:14.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-09-15T02:08:14.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:08:14.770 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:08:14.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13eba588-e279-4d17-b764-0f4d7e6a50e6 2024-09-15T02:08:14.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 13eba588-e279-4d17-b764-0f4d7e6a50e6' 2024-09-15T02:08:14.770 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 13eba588-e279-4d17-b764-0f4d7e6a50e6 2024-09-15T02:08:14.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:08:14.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOQeZmJlMFLhAAIAjJPu7OvnRz4tc11QLXGw== 2024-09-15T02:08:14.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOQeZmJlMFLhAAIAjJPu7OvnRz4tc11QLXGw=="}' 2024-09-15T02:08:14.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13eba588-e279-4d17-b764-0f4d7e6a50e6 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:08:15.043 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:08:15.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:08:15.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQCOQeZmJlMFLhAAIAjJPu7OvnRz4tc11QLXGw== --osd-uuid 13eba588-e279-4d17-b764-0f4d7e6a50e6 2024-09-15T02:08:15.076 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:15.075+0000 7f09ff727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:15.080 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:15.079+0000 7f09ff727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:15.081 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:15.080+0000 7f09ff727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:15.082 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:15.080+0000 7f09ff727440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:08:15.082 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:15.080+0000 7f09ff727440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:08:15.082 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:15.080+0000 7f09ff727440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:08:15.120 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:15.119+0000 7f09ff727440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:08:20.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:08:20.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:08:20.417 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:08:20.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:08:20.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:08:20.892 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:08:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:08:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-09-15T02:08:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:08:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:08:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:08:20.913 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:20.911+0000 7fc358218440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:20.915 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:20.913+0000 7fc358218440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:20.916 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:20.914+0000 7fc358218440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:21.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:08:21.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:08:21.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:08:21.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:08:21.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:08:21.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:21.273 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:08:21.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:08:21.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:21.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:21.429 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:21.428+0000 7fc358218440 -1 Falling back to public interface 2024-09-15T02:08:21.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:22.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:22.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:22.637 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:08:22.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:08:22.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:22.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:23.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:23.903 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:23.902+0000 7fc358218440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:08:24.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:24.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:24.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:08:24.026 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:08:24.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:24.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:24.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:25.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:25.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:25.474 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:08:25.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:08:25.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:25.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:26.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:26.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:26.885 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:08:26.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:08:26.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:26.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:28.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:28.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:28.243 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:08:28.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:08:28.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:28.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:28.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:29.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:29.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:29.724 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:08:29.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:08:29.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:29.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:30.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:31.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:31.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:31.100 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:08:31.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:08:31.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:31.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:31.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:32.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:32.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:32.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:08:32.585 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:08:32.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:32.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:32.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:33.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:33.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:33.974 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:08:33.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:08:33.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:33.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:34.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:35.361 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:08:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:08:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:36.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:37.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:37.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:37.173 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:08:37.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:08:37.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:37.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:37.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:38.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:38.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:38.756 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:08:38.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:08:38.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:38.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:39.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:40.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:40.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:40.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:08:40.449 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:08:40.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:40.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:41.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:42.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:42.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:42.255 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:08:42.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:08:42.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:42.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:43.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:44.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:44.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:44.410 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:08:44.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:08:44.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:44.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:45.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:46.031 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:08:46.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:08:46.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:46.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:46.389 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/38254,v1:127.0.0.1:6803/38254] [v2:127.0.0.1:6804/38254,v1:127.0.0.1:6805/38254] exists,up 13eba588-e279-4d17-b764-0f4d7e6a50e6 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:08:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:08:46.464 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:08:46.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:08:46.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:08:46.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:08:46.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:46.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:08:46.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:08:46.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:08:46.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:08:46.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:08:46.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:08:46.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:08:46.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:08:46.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:08:46.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:08:46.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:08:46.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:08:46.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:08:46.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:08:46.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:46.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:08:46.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:08:46.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:46.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:08:46.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:08:46.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:08:46.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:08:46.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:46.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:46.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:08:46.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:46.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:08:46.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:08:46.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:08:46.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:08:46.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:08:46.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:08:46.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:08:46.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:46.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:46.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:08:46.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:08:46.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:08:46.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:08:46.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:08:46.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:46.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:46.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:08:46.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:08:46.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:08:46.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:08:46.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:08:46.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-09-15T02:08:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:08:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:08:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:08:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:08:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-09-15T02:08:46.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:08:46.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:08:46.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:08:46.951 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:08:46.951 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:46.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:46.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:46.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-09-15T02:08:47.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:08:47.187 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:08:47.188 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:47.188 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:47.188 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:47.188 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:08:47.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:08:47.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:08:47.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:08:47.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:08:47.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:08:47.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:08:47.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:08:47.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:08:47.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:08:47.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:47.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:08:47.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:08:47.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:08:47.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:08:47.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:08:48.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:08:48.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:08:48.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:08:48.171 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:08:48.171 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:48.171 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:48.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:48.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:08:48.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:08:48.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-15T02:08:48.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:08:48.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:08:48.195 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:08:48.195 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:48.195 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:48.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:08:48.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:08:48.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:08:48.262 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:08:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:08:48.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:08:48.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:08:48.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c31f83f5-e576-4e09-bf86-727a2c244b1c 2024-09-15T02:08:48.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c31f83f5-e576-4e09-bf86-727a2c244b1c' 2024-09-15T02:08:48.276 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 c31f83f5-e576-4e09-bf86-727a2c244b1c 2024-09-15T02:08:48.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:08:48.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwQeZmWfT+EBAAZ/6qK23U4PMn6fyev9RfTw== 2024-09-15T02:08:48.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwQeZmWfT+EBAAZ/6qK23U4PMn6fyev9RfTw=="}' 2024-09-15T02:08:48.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c31f83f5-e576-4e09-bf86-727a2c244b1c -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:08:48.557 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:08:48.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:08:48.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQCwQeZmWfT+EBAAZ/6qK23U4PMn6fyev9RfTw== --osd-uuid c31f83f5-e576-4e09-bf86-727a2c244b1c 2024-09-15T02:08:48.591 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:48.589+0000 7fb084073440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:48.595 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:48.593+0000 7fb084073440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:48.596 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:48.594+0000 7fb084073440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:48.596 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:48.595+0000 7fb084073440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:08:48.596 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:48.595+0000 7fb084073440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:08:48.596 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:48.595+0000 7fb084073440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:08:48.635 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:48.634+0000 7fb084073440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:08:53.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:08:53.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:08:53.944 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:08:53.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:08:53.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:08:54.424 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:08:54.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:08:54.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-09-15T02:08:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:08:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:08:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:08:54.445 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:54.442+0000 7f5b0f433440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:54.447 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:54.445+0000 7f5b0f433440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:54.448 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:54.446+0000 7f5b0f433440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:08:54.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:08:54.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:08:54.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:08:54.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:08:54.812 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:08:54.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:08:54.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:54.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:08:54.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:54.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:54.969 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:54.967+0000 7f5b0f433440 -1 Falling back to public interface 2024-09-15T02:08:55.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:56.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:56.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:56.162 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:08:56.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:08:56.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:56.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:56.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:57.453 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:08:57.450+0000 7f5b0f433440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:08:57.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:57.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:57.521 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:08:57.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:08:57.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:57.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:58.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:08:59.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:08:59.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:08:59.030 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:08:59.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:08:59.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:08:59.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:08:59.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:00.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:00.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:00.450 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:09:00.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:09:00.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:00.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:00.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:01.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:01.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:01.855 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:09:01.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:09:01.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:01.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:02.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:03.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:03.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:03.353 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:09:03.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:09:03.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:03.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:03.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:04.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:04.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:04.737 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:09:04.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:09:04.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:04.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:05.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:06.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:06.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:06.266 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:09:06.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:09:06.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:06.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:06.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:07.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:07.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:07.620 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:09:07.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:09:07.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:07.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:07.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:08.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:08.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:08.977 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:09:08.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:09:08.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:08.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:09.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:10.616 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:09:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:09:10.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:10.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:11.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:12.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:12.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:12.316 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:09:12.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:09:12.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:12.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:13.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:14.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:14.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:14.068 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:09:14.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:09:14.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:14.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:15.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:16.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:16.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:16.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:09:16.158 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:09:16.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:16.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:16.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:17.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:17.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:17.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:09:17.942 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:09:17.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:17.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:18.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:19.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:19.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:19.463 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:09:19.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:09:19.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:19.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:19.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:20.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:20.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:20.904 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:09:20.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:09:20.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:20.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:21.299 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/40259,v1:127.0.0.1:6803/40259] [v2:127.0.0.1:6804/40259,v1:127.0.0.1:6805/40259] exists,up c31f83f5-e576-4e09-bf86-727a2c244b1c 2024-09-15T02:09:21.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-15T02:09:21.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:09:21.376 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:09:21.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:09:21.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:09:21.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:09:21.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:09:21.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:09:21.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:09:21.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:09:21.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:09:21.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:09:21.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:09:21.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:09:21.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:21.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:09:21.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:09:21.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:21.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:09:21.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:09:21.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:09:21.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:09:21.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:21.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:21.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:09:21.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:09:21.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:09:21.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:09:21.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:09:21.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:09:21.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:09:21.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:09:21.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:09:21.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:09:21.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:09:21.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:09:21.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:09:21.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:09:21.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:09:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:09:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:09:21.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:09:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:09:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:09:21.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:09:21.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:21.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:09:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:09:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:09:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:09:21.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:09:21.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:21.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:09:21.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:09:21.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:09:21.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-15T02:09:21.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:09:21.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:09:21.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:09:21.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:09:21.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:21.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:21.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:21.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:09:22.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:09:22.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:09:22.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:09:22.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:09:22.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:09:22.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:09:22.105 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:09:22.105 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:09:22.105 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:09:22.106 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:22.106 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:22.106 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:22.106 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:09:22.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:09:22.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:09:22.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:09:22.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:09:22.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:09:22.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:09:22.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:09:22.180 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:09:22.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-15T02:09:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:09:22.322 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:09:22.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:09:22.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:22.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:09:22.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:09:22.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:09:22.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:09:22.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:09:23.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:09:23.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:09:23.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:09:23.165 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:09:23.165 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:23.165 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:23.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:23.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:09:23.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:09:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-09-15T02:09:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:09:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:09:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:09:23.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:09:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:09:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:09:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:09:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:09:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:09:23.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:09:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=69e7bdac-3052-4a00-9c5b-5e0a209bddf9 2024-09-15T02:09:23.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 69e7bdac-3052-4a00-9c5b-5e0a209bddf9' 2024-09-15T02:09:23.194 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 69e7bdac-3052-4a00-9c5b-5e0a209bddf9 2024-09-15T02:09:23.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:09:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTQeZmwE8TDBAAAvhExExaf+lb47pw+Xmy7Q== 2024-09-15T02:09:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTQeZmwE8TDBAAAvhExExaf+lb47pw+Xmy7Q=="}' 2024-09-15T02:09:23.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 69e7bdac-3052-4a00-9c5b-5e0a209bddf9 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:09:23.509 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:09:23.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:09:23.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTQeZmwE8TDBAAAvhExExaf+lb47pw+Xmy7Q== --osd-uuid 69e7bdac-3052-4a00-9c5b-5e0a209bddf9 2024-09-15T02:09:23.541 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:23.540+0000 7fcf70a67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:23.563 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:23.562+0000 7fcf70a67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:23.564 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:23.563+0000 7fcf70a67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:23.565 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:23.563+0000 7fcf70a67440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:09:23.565 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:23.563+0000 7fcf70a67440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:09:23.565 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:23.563+0000 7fcf70a67440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:09:23.603 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:23.602+0000 7fcf70a67440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:09:28.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:09:28.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:09:28.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:09:28.925 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:09:28.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:09:29.401 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:09:29.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:09:29.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:09:29.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:09:29.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:09:29.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:09:29.422 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:29.420+0000 7ff38ad22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:29.424 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:29.422+0000 7ff38ad22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:29.425 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:29.423+0000 7ff38ad22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:09:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:09:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:09:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:09:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:09:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:09:29.747 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:09:29.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:29.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:29.945 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:29.944+0000 7ff38ad22440 -1 Falling back to public interface 2024-09-15T02:09:30.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:31.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:31.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:31.107 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:09:31.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:09:31.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:31.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:31.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:32.413 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:32.412+0000 7ff38ad22440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:09:32.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:32.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:32.461 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:09:32.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:09:32.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:32.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:33.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:34.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:34.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:34.006 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:09:34.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:09:34.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:34.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:34.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:35.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:35.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:35.478 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:09:35.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:09:35.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:35.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:35.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:36.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:36.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:36.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:09:36.856 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:09:36.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:36.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:37.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:38.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:38.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:38.245 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:09:38.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:09:38.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:38.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:38.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:39.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:39.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:39.612 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:09:39.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:09:39.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:39.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:40.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:41.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:41.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:41.116 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:09:41.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:09:41.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:41.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:41.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:42.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:42.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:42.477 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:09:42.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:09:42.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:42.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:42.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:43.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:43.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:43.832 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:09:43.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:09:43.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:43.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:44.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:45.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:45.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:45.730 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:09:45.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:09:45.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:45.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:46.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:47.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:47.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:47.278 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:09:47.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:09:47.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:47.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:48.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:49.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:49.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:49.049 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:09:49.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:09:49.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:49.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:49.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:50.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:50.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:50.719 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:09:50.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:09:50.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:50.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:51.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:52.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:52.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:52.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:09:52.318 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:09:52.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:52.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:53.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:54.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:54.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:54.061 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:09:54.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:09:54.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:54.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:54.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:09:55.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:09:55.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:09:55.693 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:09:55.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:09:55.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:09:55.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:09:56.045 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/42301,v1:127.0.0.1:6803/42301] [v2:127.0.0.1:6804/42301,v1:127.0.0.1:6805/42301] exists,up 69e7bdac-3052-4a00-9c5b-5e0a209bddf9 2024-09-15T02:09:56.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:09:56.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:09:56.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:09:56.046 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-09-15T02:09:56.046 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:09:56.046 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:09:56.046 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:56.047 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:56.047 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:56.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:09:56.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-15T02:09:56.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:09:56.120 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:09:56.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:09:56.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:09:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-09-15T02:09:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:09:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:09:56.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:09:56.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:09:56.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:09:56.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:09:56.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:09:56.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:09:56.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:09:56.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:09:56.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:09:56.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:09:56.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:09:56.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:09:56.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:56.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:09:56.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:09:56.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:56.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:09:56.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:09:56.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:09:56.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:09:56.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:56.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:56.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:09:56.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:09:56.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:09:56.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:09:56.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:09:56.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:09:56.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:09:56.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:09:56.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:09:56.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:09:56.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:09:56.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:09:56.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:09:56.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:09:56.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:09:56.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:56.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:09:56.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:09:56.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:09:56.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:09:56.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:09:56.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:09:56.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:09:56.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:56.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:56.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:09:56.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:09:56.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:09:56.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:09:56.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:09:56.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:56.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:56.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:09:56.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-09-15T02:09:56.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:09:56.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:09:56.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:09:56.602 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:09:56.602 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:56.602 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:56.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:56.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-09-15T02:09:56.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:09:56.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:09:56.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:09:56.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:09:56.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:09:56.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:09:56.846 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:09:56.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:09:56.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:09:56.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:09:56.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:09:56.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:09:56.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:09:56.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:09:56.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:56.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:09:56.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:09:56.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:09:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:09:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:57.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:09:57.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:09:57.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:09:57.920 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:09:57.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-09-15T02:09:57.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:09:57.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:09:57.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:09:57.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:09:57.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:09:57.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:09:57.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-09-15T02:09:57.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:09:57.933 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:09:57.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=872ebfac-56a1-4338-8eea-727c49374a0f 2024-09-15T02:09:57.934 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 872ebfac-56a1-4338-8eea-727c49374a0f 2024-09-15T02:09:57.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 872ebfac-56a1-4338-8eea-727c49374a0f' 2024-09-15T02:09:57.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:09:57.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1QeZmoTEwOBAAO17HgdheGEbhyntsU8EZrg== 2024-09-15T02:09:57.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1QeZmoTEwOBAAO17HgdheGEbhyntsU8EZrg=="}' 2024-09-15T02:09:57.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 872ebfac-56a1-4338-8eea-727c49374a0f -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:09:58.241 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:09:58.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:09:58.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQD1QeZmoTEwOBAAO17HgdheGEbhyntsU8EZrg== --osd-uuid 872ebfac-56a1-4338-8eea-727c49374a0f 2024-09-15T02:09:58.274 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:58.272+0000 7f4f45c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:58.295 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:58.293+0000 7f4f45c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:58.296 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:58.294+0000 7f4f45c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:09:58.296 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:58.295+0000 7f4f45c0d440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:09:58.296 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:58.295+0000 7f4f45c0d440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:09:58.296 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:58.295+0000 7f4f45c0d440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:09:58.335 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:09:58.333+0000 7f4f45c0d440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:10:03.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:10:03.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:10:03.639 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:10:03.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:10:03.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:10:04.115 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:10:04.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:10:04.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-09-15T02:10:04.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:10:04.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:10:04.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:10:04.137 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:04.135+0000 7fd3cbf22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:04.138 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:04.137+0000 7fd3cbf22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:04.139 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:04.138+0000 7fd3cbf22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:04.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:10:04.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:10:04.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:10:04.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:10:04.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:10:04.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:04.495 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:10:04.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:10:04.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:04.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:04.654 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:04.653+0000 7fd3cbf22440 -1 Falling back to public interface 2024-09-15T02:10:04.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:05.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:05.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:05.856 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:10:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:10:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:05.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:06.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:07.150 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:07.149+0000 7fd3cbf22440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:10:07.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:07.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:07.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:10:07.210 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:10:07.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:07.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:07.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:08.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:08.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:08.716 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:10:08.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:10:08.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:08.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:09.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:10.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:10.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:10.099 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:10:10.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:10:10.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:10.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:10.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:11.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:11.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:11.487 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:10:11.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:10:11.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:11.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:11.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:12.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:12.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:12.977 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:10:12.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:10:12.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:12.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:13.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:14.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:14.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:14.336 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:10:14.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:10:14.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:14.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:14.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:15.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:15.848 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:10:15.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:15.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:10:15.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:15.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:16.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:17.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:17.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:17.226 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:10:17.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:10:17.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:17.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:17.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:18.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:18.586 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:10:18.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:18.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:10:18.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:18.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:19.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:20.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:20.502 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:10:20.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:20.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:10:20.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:20.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:21.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:22.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:22.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:22.010 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:10:22.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:10:22.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:22.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:22.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:23.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:23.631 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:10:23.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:23.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:10:23.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:23.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:24.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:25.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:25.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:25.686 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:10:25.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:10:25.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:25.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:26.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:27.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:27.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:27.438 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:10:27.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:10:27.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:27.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:28.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:29.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:29.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:29.268 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:10:29.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:10:29.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:29.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:29.686 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/44332,v1:127.0.0.1:6803/44332] [v2:127.0.0.1:6804/44332,v1:127.0.0.1:6805/44332] exists,up 872ebfac-56a1-4338-8eea-727c49374a0f 2024-09-15T02:10:29.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:10:29.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:10:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:10:29.761 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:10:29.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:10:29.770 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:09:56.623+0000 7fa2ad4cd900 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T02:10:29.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:10:29.772 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:10:04.654+0000 7fd3cbf22440 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:10:29.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:10:29.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:10:29.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:10:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:10:29.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:10:29.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:10:29.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:10:29.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:10:29.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:10:29.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:10:29.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:10:29.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:10:29.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:10:29.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:10:29.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:10:29.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:10:29.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:29.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:29.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:10:29.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:10:29.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:10:29.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:10:29.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:10:29.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:10:29.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:10:29.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:10:29.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:10:29.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:10:29.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:10:29.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:10:29.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:10:29.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:10:29.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:10:29.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:10:29.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:10:29.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:10:29.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:10:29.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:10:29.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:29.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:29.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:10:29.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:10:29.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:10:29.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:10:29.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:10:29.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:29.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:29.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:10:29.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T02:10:30.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:10:30.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:10:30.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:10:30.257 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:10:30.257 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:30.257 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:30.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:10:30.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T02:10:30.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:10:30.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:10:30.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:10:30.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:10:30.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:10:30.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:10:30.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:10:30.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:10:30.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:10:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:10:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:10:30.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:10:30.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:10:30.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:10:30.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:10:30.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:10:30.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:10:30.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:10:30.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:10:30.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:10:30.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:10:30.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:10:30.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:10:30.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:10:30.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:10:30.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:10:31.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:10:31.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:10:31.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:10:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:10:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:31.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:10:31.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:10:31.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:10:31.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T02:10:31.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:10:31.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:10:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:10:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:31.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:10:31.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:10:31.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:10:31.580 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:10:31.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:10:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:10:31.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:10:31.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a05dd685-2fae-4ff7-b750-49b9dc43bd07 2024-09-15T02:10:31.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a05dd685-2fae-4ff7-b750-49b9dc43bd07' 2024-09-15T02:10:31.591 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 a05dd685-2fae-4ff7-b750-49b9dc43bd07 2024-09-15T02:10:31.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:10:31.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXQuZm/0zIIxAA0fvSd0IMyd3VyV7v44xdIw== 2024-09-15T02:10:31.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXQuZm/0zIIxAA0fvSd0IMyd3VyV7v44xdIw=="}' 2024-09-15T02:10:31.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a05dd685-2fae-4ff7-b750-49b9dc43bd07 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:10:31.872 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:10:31.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:10:31.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQAXQuZm/0zIIxAA0fvSd0IMyd3VyV7v44xdIw== --osd-uuid a05dd685-2fae-4ff7-b750-49b9dc43bd07 2024-09-15T02:10:31.905 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:31.904+0000 7f9758611440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:31.917 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:31.916+0000 7f9758611440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:31.918 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:31.917+0000 7f9758611440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:31.918 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:31.917+0000 7f9758611440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:10:31.918 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:31.917+0000 7f9758611440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:10:31.918 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:31.917+0000 7f9758611440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:10:31.957 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:31.956+0000 7f9758611440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:10:37.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:10:37.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:10:37.268 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:10:37.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:10:37.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:10:37.747 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:10:37.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:10:37.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-09-15T02:10:37.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:10:37.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:10:37.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:10:37.769 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:37.767+0000 7f56f191d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:37.771 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:37.769+0000 7f56f191d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:37.772 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:37.771+0000 7f56f191d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:10:38.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:10:38.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:10:38.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:10:38.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:10:38.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:10:38.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:38.095 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:10:38.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:10:38.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:38.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:38.287 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:38.286+0000 7f56f191d440 -1 Falling back to public interface 2024-09-15T02:10:38.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:39.452 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:10:39.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:10:39.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:39.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:39.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:40.766 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:10:40.764+0000 7f56f191d440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:10:40.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:40.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:40.809 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:10:40.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:10:40.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:40.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:41.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:42.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:42.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:42.345 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:10:42.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:10:42.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:42.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:42.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:43.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:43.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:43.739 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:10:43.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:10:43.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:43.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:44.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:45.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:45.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:45.111 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:10:45.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:10:45.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:45.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:45.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:46.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:46.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:46.536 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:10:46.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:10:46.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:46.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:46.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:47.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:47.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:47.929 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:10:47.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:10:47.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:47.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:48.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:49.290 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:10:49.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:10:49.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:49.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:49.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:50.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:50.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:50.738 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:10:50.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:10:50.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:50.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:51.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:52.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:52.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:52.127 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:10:52.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:10:52.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:52.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:52.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:53.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:53.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:53.983 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:10:53.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:10:53.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:53.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:55.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:55.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:55.711 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:10:55.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:10:55.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:55.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:56.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:57.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:57.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:57.431 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:10:57.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:10:57.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:57.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:58.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:10:59.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:10:59.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:10:59.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:10:59.456 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:10:59.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:10:59.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:10:59.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:00.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:00.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:00.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:11:00.889 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:11:00.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:00.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:01.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:02.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:02.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:02.372 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:11:02.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:11:02.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:02.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:02.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:11:03.752 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:11:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:04.112 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/46337,v1:127.0.0.1:6803/46337] [v2:127.0.0.1:6804/46337,v1:127.0.0.1:6805/46337] exists,up a05dd685-2fae-4ff7-b750-49b9dc43bd07 2024-09-15T02:11:04.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:11:04.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:11:04.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:11:04.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T02:11:04.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:11:04.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:11:04.114 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:04.114 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:04.114 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:04.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:11:04.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:11:04.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:11:04.187 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:11:04.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:11:04.196 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:10:30.278+0000 7fa7968c9900 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T02:11:04.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:11:04.197 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:10:38.287+0000 7f56f191d440 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T02:11:04.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:11:04.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:04.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:11:04.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:11:04.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:11:04.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:11:04.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:11:04.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:11:04.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:11:04.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:11:04.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:11:04.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:11:04.305 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:11:04.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:11:04.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:11:04.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:11:04.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:04.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:11:04.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:11:04.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:04.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:11:04.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:11:04.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:11:04.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:11:04.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:04.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:04.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:11:04.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:11:04.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:11:04.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T02:11:04.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T02:11:04.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:11:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:11:04.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:11:04.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:11:04.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:11:04.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:11:04.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:11:04.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:11:04.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:11:04.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:04.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:11:04.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:11:04.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:04.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:11:04.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:11:04.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:11:04.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:11:04.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:04.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:04.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:11:04.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:11:04.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:11:04.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:11:04.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:11:04.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:04.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:04.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:11:04.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:11:04.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:11:04.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:11:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T02:11:04.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:11:04.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:11:04.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:11:04.717 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:11:04.717 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:04.718 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:04.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:04.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T02:11:04.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:11:04.936 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:04.937 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:04.937 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:04.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:11:04.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:11:04.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:11:05.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:11:05.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:11:05.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:11:05.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:11:05.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:05.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:11:05.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:11:05.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:11:05.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:11:05.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:11:05.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:11:05.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:11:05.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:11:05.921 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:11:05.921 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:05.921 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:05.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:05.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:11:05.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:11:05.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T02:11:05.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:11:05.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:11:05.944 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:05.944 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:05.944 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:05.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:11:05.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:11:05.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:11:06.015 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:11:06.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T02:11:06.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:06.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:11:06.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:11:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:06.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:11:06.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:11:06.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:11:06.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:11:06.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-09-15T02:11:06.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:11:06.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:11:06.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e8bebc6-7dfc-48cb-977b-cd5e80c7f56c 2024-09-15T02:11:06.031 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 3e8bebc6-7dfc-48cb-977b-cd5e80c7f56c 2024-09-15T02:11:06.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3e8bebc6-7dfc-48cb-977b-cd5e80c7f56c' 2024-09-15T02:11:06.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:11:06.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6QuZmERw/AhAAu0ZcZuXhB21bgb6TvY1T/w== 2024-09-15T02:11:06.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6QuZmERw/AhAAu0ZcZuXhB21bgb6TvY1T/w=="}' 2024-09-15T02:11:06.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e8bebc6-7dfc-48cb-977b-cd5e80c7f56c -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:11:06.315 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:11:06.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:11:06.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQA6QuZmERw/AhAAu0ZcZuXhB21bgb6TvY1T/w== --osd-uuid 3e8bebc6-7dfc-48cb-977b-cd5e80c7f56c 2024-09-15T02:11:06.348 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:06.347+0000 7f4456b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:06.360 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:06.359+0000 7f4456b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:06.361 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:06.360+0000 7f4456b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:06.361 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:06.360+0000 7f4456b1f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:11:06.362 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:06.360+0000 7f4456b1f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:11:06.362 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:06.360+0000 7f4456b1f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:11:06.400 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:06.399+0000 7f4456b1f440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:11:11.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:11:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:11:11.728 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:11:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:11:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:11:12.219 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:11:12.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:11:12.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-09-15T02:11:12.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:11:12.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:11:12.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:11:12.240 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:12.239+0000 7fc987616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:12.242 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:12.241+0000 7fc987616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:12.243 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:12.242+0000 7fc987616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:12.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:11:12.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:11:12.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:11:12.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:11:12.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:11:12.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:12.564 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:11:12.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:11:12.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:12.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:12.758 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:12.757+0000 7fc987616440 -1 Falling back to public interface 2024-09-15T02:11:12.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:13.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:13.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:13.919 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:11:13.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:11:13.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:13.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:14.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:15.248 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:15.246+0000 7fc987616440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:11:15.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:15.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:15.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:11:15.320 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:11:15.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:15.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:15.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:16.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:16.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:16.720 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:11:16.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:11:16.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:16.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:17.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:18.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:18.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:18.080 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:11:18.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:11:18.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:18.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:19.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:19.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:19.490 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:11:19.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:11:19.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:19.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:19.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:20.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:20.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:20.944 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:11:20.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:11:20.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:20.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:21.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:22.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:22.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:22.301 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:11:22.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:11:22.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:22.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:22.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:23.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:23.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:23.688 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:11:23.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:11:23.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:23.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:24.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:25.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:25.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:25.085 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:11:25.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:11:25.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:25.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:25.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:26.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:26.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:26.482 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:11:26.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:11:26.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:26.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:26.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:27.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:27.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:27.888 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:11:27.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:11:27.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:27.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:28.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:29.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:29.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:29.358 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:11:29.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:11:29.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:29.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:30.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:31.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:31.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:31.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:11:31.350 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:11:31.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:31.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:32.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:33.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:33.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:33.231 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:11:33.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:11:33.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:33.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:33.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:34.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:34.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:34.885 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:11:34.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:11:34.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:34.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:35.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:36.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:36.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:36.745 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:11:36.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:11:36.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:36.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:37.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:38.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:38.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:38.257 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:11:38.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:11:38.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:38.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:38.644 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/48366,v1:127.0.0.1:6803/48366] [v2:127.0.0.1:6804/48366,v1:127.0.0.1:6805/48366] exists,up 3e8bebc6-7dfc-48cb-977b-cd5e80c7f56c 2024-09-15T02:11:38.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:11:38.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:11:38.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:11:38.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T02:11:38.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:11:38.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:11:38.646 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:38.647 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:38.647 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:38.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:11:38.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:11:38.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:11:38.719 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:11:38.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:11:38.729 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:11:04.739+0000 7f24c52d0900 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T02:11:38.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:11:38.730 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:11:12.758+0000 7fc987616440 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-15T02:11:38.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:11:38.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:38.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:11:38.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:11:38.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:11:38.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:11:38.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:11:38.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:11:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:11:38.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:11:38.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:11:38.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:11:38.838 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:11:38.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:11:38.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:11:38.839 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:11:38.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:38.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:11:38.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:11:38.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:38.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:11:38.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:11:38.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:11:38.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:11:38.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:38.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:38.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:11:38.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:11:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:11:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T02:11:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T02:11:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:11:38.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:38.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:11:38.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:11:38.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:11:38.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:11:38.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:11:38.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:11:38.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:11:38.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:11:38.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:11:38.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:11:38.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:11:38.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:11:38.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:11:38.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:11:38.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:38.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:11:38.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:11:38.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:11:38.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:11:38.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:11:38.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:11:38.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:11:38.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:38.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:38.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:11:38.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:11:38.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:11:38.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:11:38.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:11:38.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:38.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:38.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:11:38.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:11:38.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:11:38.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:11:38.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:11:38.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-09-15T02:11:38.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:38.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:11:38.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:11:38.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:11:38.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:11:38.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-09-15T02:11:39.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:11:39.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:11:39.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:11:39.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:11:39.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:39.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:39.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:39.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-09-15T02:11:39.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:11:39.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:11:39.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:11:39.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:11:39.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:11:39.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:11:39.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:11:39.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:11:39.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:11:39.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:11:39.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:11:39.526 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:11:39.526 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:11:39.527 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:11:39.527 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:39.527 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:39.527 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:39.527 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:11:39.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:11:39.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:11:39.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:11:39.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:39.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:11:39.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:11:39.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:11:39.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:11:39.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:11:40.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:11:40.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:11:40.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:11:40.436 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:11:40.436 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:40.436 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:40.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:40.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:11:40.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:11:40.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T02:11:40.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:11:40.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:11:40.461 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:11:40.462 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:40.462 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:40.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:11:40.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:11:40.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:11:40.527 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:11:40.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-09-15T02:11:40.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:11:40.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:11:40.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:11:40.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:11:40.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:11:40.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:11:40.539 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:11:40.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:11:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-09-15T02:11:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:11:40.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:11:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b442e26-9e9d-4406-861d-03f7557be513 2024-09-15T02:11:40.541 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 8b442e26-9e9d-4406-861d-03f7557be513 2024-09-15T02:11:40.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8b442e26-9e9d-4406-861d-03f7557be513' 2024-09-15T02:11:40.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:11:40.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcQuZmMZLMIBAAo0jziXTVlwpsDH8Iv0KbXA== 2024-09-15T02:11:40.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcQuZmMZLMIBAAo0jziXTVlwpsDH8Iv0KbXA=="}' 2024-09-15T02:11:40.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b442e26-9e9d-4406-861d-03f7557be513 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:11:40.820 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:11:40.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:11:40.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQBcQuZmMZLMIBAAo0jziXTVlwpsDH8Iv0KbXA== --osd-uuid 8b442e26-9e9d-4406-861d-03f7557be513 2024-09-15T02:11:40.854 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:40.852+0000 7f5696a3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:40.865 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:40.864+0000 7f5696a3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:40.866 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:40.865+0000 7f5696a3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:40.867 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:40.866+0000 7f5696a3b440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:11:40.867 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:40.866+0000 7f5696a3b440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:11:40.867 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:40.866+0000 7f5696a3b440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:11:40.906 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:40.904+0000 7f5696a3b440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:11:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:11:46.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:11:46.228 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:11:46.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:11:46.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:11:46.707 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:11:46.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:11:46.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-09-15T02:11:46.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:11:46.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:11:46.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:11:46.728 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:46.726+0000 7fcb5a71d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:46.730 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:46.728+0000 7fcb5a71d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:46.731 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:46.729+0000 7fcb5a71d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:11:47.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:11:47.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:11:47.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:11:47.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:11:47.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:11:47.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:47.054 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:11:47.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:11:47.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:47.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:47.251 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:47.249+0000 7fcb5a71d440 -1 Falling back to public interface 2024-09-15T02:11:47.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:48.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:48.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:48.416 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:11:48.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:11:48.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:48.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:48.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:49.726 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:11:49.723+0000 7fcb5a71d440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:11:49.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:49.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:49.770 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:11:49.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:11:49.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:49.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:50.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:51.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:51.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:51.295 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:11:51.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:11:51.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:51.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:51.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:52.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:52.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:52.718 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:11:52.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:11:52.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:52.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:53.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:54.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:54.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:54.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:11:54.103 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:11:54.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:54.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:54.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:55.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:55.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:55.593 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:11:55.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:11:55.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:55.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:55.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:56.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:56.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:56.968 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:11:56.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:11:56.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:56.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:57.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:58.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:58.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:58.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:11:58.412 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:11:58.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:58.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:11:58.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:11:59.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:11:59.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:11:59.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:11:59.771 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:11:59.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:11:59.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:00.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:01.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:01.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:01.128 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:12:01.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:12:01.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:01.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:01.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:02.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:02.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:02.742 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:12:02.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:12:02.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:02.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:03.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:04.418 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:12:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:12:04.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:04.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:05.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:06.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:06.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:06.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:12:06.241 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:12:06.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:06.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:07.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:08.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:08.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:08.328 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:12:08.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:12:08.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:08.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:08.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:09.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:09.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:09.783 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:12:09.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:12:09.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:09.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:10.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:11.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:11.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:11.310 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:12:11.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:12:11.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:11.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:11.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:12.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:12.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:12.813 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:12:12.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:12:12.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:12.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:13.182 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/50395,v1:127.0.0.1:6803/50395] [v2:127.0.0.1:6804/50395,v1:127.0.0.1:6805/50395] exists,up 8b442e26-9e9d-4406-861d-03f7557be513 2024-09-15T02:12:13.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:12:13.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:12:13.257 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:12:13.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:12:13.266 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:11:39.238+0000 7f44fb25d900 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T02:12:13.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:12:13.268 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:11:47.250+0000 7fcb5a71d440 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T02:12:13.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:12:13.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:13.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:12:13.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:12:13.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:12:13.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:12:13.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:12:13.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:12:13.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:12:13.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:12:13.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:12:13.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:12:13.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:12:13.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:12:13.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:12:13.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:12:13.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:13.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:12:13.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:12:13.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:13.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:12:13.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:12:13.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:12:13.385 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:12:13.385 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:13.385 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:13.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:12:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:12:13.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:12:13.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:12:13.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:12:13.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:12:13.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:12:13.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:12:13.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:12:13.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:12:13.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:12:13.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:12:13.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:12:13.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:12:13.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:13.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:12:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:12:13.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:12:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:12:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:12:13.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:12:13.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:13.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:13.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:12:13.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:12:13.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:12:13.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:12:13.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:12:13.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:13.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:13.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:12:13.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:12:13.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:12:13.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:12:13.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:12:13.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-09-15T02:12:13.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:13.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:12:13.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:12:13.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:12:13.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:12:13.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-09-15T02:12:13.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:12:13.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:12:13.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:12:13.753 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:12:13.753 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:13.753 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:13.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:13.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-09-15T02:12:14.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:12:14.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:12:14.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:12:14.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:12:14.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:12:14.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:12:14.005 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:12:14.005 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:12:14.005 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:12:14.006 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:14.007 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:14.007 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:14.007 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:12:14.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:12:14.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:12:14.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:12:14.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:12:14.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:12:14.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:12:14.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:12:14.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:12:14.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:12:14.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:12:14.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:14.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:12:14.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:12:14.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:12:14.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:12:14.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:12:14.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:12:14.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:12:14.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:12:14.990 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:12:14.990 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:14.990 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:14.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:14.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:12:14.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:12:15.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T02:12:15.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:12:15.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:12:15.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:15.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:15.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:15.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:12:15.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:12:15.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:12:15.081 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:12:15.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-09-15T02:12:15.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:15.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:12:15.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:12:15.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:12:15.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:12:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:12:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:12:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:12:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:12:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-09-15T02:12:15.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:12:15.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:12:15.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d26df7f5-18bb-4b3c-af66-b7a7fcdbc1cc 2024-09-15T02:12:15.094 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 d26df7f5-18bb-4b3c-af66-b7a7fcdbc1cc 2024-09-15T02:12:15.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d26df7f5-18bb-4b3c-af66-b7a7fcdbc1cc' 2024-09-15T02:12:15.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:12:15.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/QuZmumIqBhAAgdM8i7HlAAfVXaMFHhuLNg== 2024-09-15T02:12:15.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/QuZmumIqBhAAgdM8i7HlAAfVXaMFHhuLNg=="}' 2024-09-15T02:12:15.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d26df7f5-18bb-4b3c-af66-b7a7fcdbc1cc -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:12:15.374 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:12:15.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:12:15.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQB/QuZmumIqBhAAgdM8i7HlAAfVXaMFHhuLNg== --osd-uuid d26df7f5-18bb-4b3c-af66-b7a7fcdbc1cc 2024-09-15T02:12:15.408 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:15.406+0000 7f667941f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:15.421 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:15.420+0000 7f667941f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:15.422 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:15.421+0000 7f667941f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:15.422 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:15.421+0000 7f667941f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:12:15.422 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:15.421+0000 7f667941f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:12:15.423 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:15.421+0000 7f667941f440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:12:15.461 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:15.460+0000 7f667941f440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:12:20.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:12:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:12:20.774 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:12:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:12:20.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:12:21.254 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:12:21.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:12:21.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-09-15T02:12:21.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:12:21.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:12:21.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:12:21.275 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:21.274+0000 7f329194c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:21.277 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:21.276+0000 7f329194c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:21.278 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:21.277+0000 7f329194c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:21.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:12:21.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:12:21.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:12:21.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:12:21.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:12:21.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:21.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:12:21.599 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:12:21.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:21.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:21.798 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:21.797+0000 7f329194c440 -1 Falling back to public interface 2024-09-15T02:12:21.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:22.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:22.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:22.962 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:12:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:12:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:23.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:24.266 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:24.265+0000 7f329194c440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:12:24.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:24.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:24.320 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:12:24.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:12:24.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:24.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:24.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:12:25.884 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:12:25.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:25.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:26.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:27.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:27.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:27.259 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:12:27.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:12:27.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:27.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:27.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:28.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:28.624 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:12:28.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:12:28.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:28.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:29.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:30.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:30.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:30.105 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:12:30.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:12:30.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:30.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:30.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:12:31.486 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:12:31.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:31.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:31.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:32.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:32.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:32.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:12:32.892 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:12:32.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:32.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:33.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:34.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:34.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:34.279 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:12:34.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:12:34.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:34.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:34.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:35.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:35.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:35.639 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:12:35.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:12:35.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:35.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:36.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:37.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:37.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:37.202 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:12:37.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:12:37.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:37.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:37.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:38.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:38.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:38.857 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:12:38.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:12:38.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:38.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:39.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:40.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:40.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:40.928 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:12:40.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:12:40.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:40.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:41.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:42.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:42.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:42.608 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:12:42.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:12:42.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:42.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:43.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:44.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:44.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:44.308 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:12:44.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:12:44.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:44.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:45.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:46.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:46.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:46.459 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:12:46.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:12:46.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:46.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:46.873 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/52425,v1:127.0.0.1:6803/52425] [v2:127.0.0.1:6804/52425,v1:127.0.0.1:6805/52425] exists,up d26df7f5-18bb-4b3c-af66-b7a7fcdbc1cc 2024-09-15T02:12:46.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:12:46.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:12:46.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:12:46.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T02:12:46.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:12:46.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:12:46.875 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:46.875 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:46.875 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:46.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:12:46.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:12:46.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:12:46.947 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:12:46.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:12:46.957 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:12:13.776+0000 7f6c52ec8900 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T02:12:46.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:12:46.958 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:12:21.798+0000 7f329194c440 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:12:46.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:12:47.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:12:47.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:12:47.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:12:47.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:12:47.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:12:47.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:12:47.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:12:47.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:47.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:12:47.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:12:47.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:47.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:12:47.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:12:47.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:12:47.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:12:47.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:47.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:47.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:12:47.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:12:47.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:12:47.079 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:12:47.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:12:47.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:12:47.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:12:47.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:12:47.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:12:47.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:12:47.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:12:47.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:12:47.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:12:47.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:12:47.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:47.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:12:47.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:12:47.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:12:47.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:12:47.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:12:47.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:12:47.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:12:47.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:47.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:47.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:12:47.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:12:47.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:12:47.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-15T02:12:47.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:12:47.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:47.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:47.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26991 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-15T02:12:47.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-09-15T02:12:47.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:12:47.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:12:47.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:12:47.452 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:12:47.452 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:47.453 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:47.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:47.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-09-15T02:12:47.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:12:47.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:12:47.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:12:47.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:12:47.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:12:47.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:12:47.654 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:12:47.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get fsid 2024-09-15T02:12:47.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:12:47.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:12:47.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:47.730 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:12:47.731 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:12:47.731 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26991/ceph-mon.a.asok config get mon_host 2024-09-15T02:12:47.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-15T02:12:47.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:47.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:12:47.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:12:47.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:12:47.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-15T02:12:47.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:12:48.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:12:48.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:12:48.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:12:48.638 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:12:48.638 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:48.638 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:48.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:48.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:12:48.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:12:48.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-15T02:12:48.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:12:48.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:12:48.663 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:12:48.663 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:48.663 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:48.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-mon.a.asok 2024-09-15T02:12:48.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:12:48.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-mon.a.asok log flush 2024-09-15T02:12:48.731 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:12:48.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:12:48.742 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:12:48.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-09-15T02:12:48.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-15T02:12:48.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:12:48.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a858fc44-8853-480c-8296-f298e3983d04 2024-09-15T02:12:48.744 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 a858fc44-8853-480c-8296-f298e3983d04 2024-09-15T02:12:48.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a858fc44-8853-480c-8296-f298e3983d04' 2024-09-15T02:12:48.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:12:48.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgQuZmhwbnLBAAXDkQos0aBYvaj483M7QVMA== 2024-09-15T02:12:48.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgQuZmhwbnLBAAXDkQos0aBYvaj483M7QVMA=="}' 2024-09-15T02:12:48.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a858fc44-8853-480c-8296-f298e3983d04 -i td/test-erasure-code-plugins/0/new.json 2024-09-15T02:12:49.024 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:12:49.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-15T02:12:49.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQCgQuZmhwbnLBAAXDkQos0aBYvaj483M7QVMA== --osd-uuid a858fc44-8853-480c-8296-f298e3983d04 2024-09-15T02:12:49.058 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:49.056+0000 7fd001e3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:49.070 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:49.068+0000 7fd001e3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:49.071 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:49.069+0000 7fd001e3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:49.072 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:49.070+0000 7fd001e3b440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:12:49.072 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:49.070+0000 7fd001e3b440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:12:49.072 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:49.070+0000 7fd001e3b440 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-15T02:12:49.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:49.132+0000 7fd001e3b440 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-15T02:12:54.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-15T02:12:54.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:12:54.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:12:54.437 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:12:54.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:12:54.912 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d7452951-051b-4cd3-829b-f28ca08f2c11 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.26991/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-09-15T02:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:12:54.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:12:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:54.933+0000 7f317d609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:54.936 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:54.935+0000 7f317d609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:54.937 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:54.936+0000 7f317d609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:12:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:12:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:12:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:12:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:12:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:12:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:55.258 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:12:55.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:12:55.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:55.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:55.466 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:55.465+0000 7f317d609440 -1 Falling back to public interface 2024-09-15T02:12:55.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:56.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:56.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:56.616 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:12:56.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:12:56.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:56.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:56.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:57.928 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:12:57.926+0000 7f317d609440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:12:57.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:57.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:57.969 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:12:57.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:12:57.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:57.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:58.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:12:59.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:12:59.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:12:59.505 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:12:59.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:12:59.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:12:59.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:12:59.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:00.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:00.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:00.921 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:13:00.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:13:00.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:00.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:01.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:02.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:02.279 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:13:02.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:02.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:13:02.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:02.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:02.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:03.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:03.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:03.743 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:13:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:13:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:04.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:05.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:05.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:05.148 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:13:05.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:13:05.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:05.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:05.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:06.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:06.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:06.602 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:13:06.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:13:06.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:06.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:06.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:07.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:07.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:07.989 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:13:07.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:13:07.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:07.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:08.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:09.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:09.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:09.372 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:13:09.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:13:09.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:09.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:10.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:11.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:11.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:11.278 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:13:11.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:13:11.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:11.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:12.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:13.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:13.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:13:13.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:13:13.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:13.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:14.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:15.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:15.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:15.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:13:15.207 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:13:15.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:15.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:16.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:17.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:17.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:17.022 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:13:17.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:13:17.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:17.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:18.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:13:19.022 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:13:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:19.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:20.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:20.872 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:13:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:13:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:21.231 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/54429,v1:127.0.0.1:6803/54429] [v2:127.0.0.1:6804/54429,v1:127.0.0.1:6805/54429] exists,up a858fc44-8853-480c-8296-f298e3983d04 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26991/ceph-osd.0.asok 2024-09-15T02:13:21.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-15T02:13:21.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.26991/ceph-osd.0.asok log flush 2024-09-15T02:13:21.306 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:13:21.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-15T02:13:21.315 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:12:47.473+0000 7f1746661900 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T02:13:21.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-15T02:13:21.317 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:12:55.466+0000 7f317d609440 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:13:21.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:13:21.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:13:21.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:13:21.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:13:21.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:13:21.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:13:21.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:13:21.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:13:21.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:13:21.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:13:21.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:13:21.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:13:21.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:13:21.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:13:21.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:13:21.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:13:21.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:21.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:13:21.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:13:21.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:13:21.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:13:21.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-09-15T02:13:21.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T02:13:21.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-erasure-code-plugins 0 2024-09-15T02:13:21.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-15T02:13:21.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T02:13:21.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-15T02:13:21.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:13:21.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:13:21.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:13:21.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:13:21.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:13:21.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:13:21.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:13:21.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:13:21.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:13:21.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T02:13:21.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-15T02:13:21.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:13:21.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:21.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26991 2024-09-15T02:13:21.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26991 2024-09-15T02:13:21.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:13:21.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:13:21.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T02:13:21.449 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:13:21.450 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:13:21.477 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-09-15T02:13:21.477 DEBUG:teuthology.orchestra.run.smithi130:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-09-15T02:13:21.539 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:13:21.546 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:13:21.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:13:21.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:13:21.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T02:13:21.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-erasure-code 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none ' 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:13:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:13:21.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:13:21.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:13:21.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:13:21.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:13:21.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:13:21.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:13:21.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:13:21.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:13:21.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:13:21.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:13:21.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:13:21.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:13:21.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:13:21.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:13:21.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-15T02:13:21.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:13:21.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:21.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:21.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55401 2024-09-15T02:13:21.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:13:21.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:13:21.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2024-09-15T02:13:21.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:13:21.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:21.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:21.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.55401 2024-09-15T02:13:21.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:13:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:13:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:13:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:13:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-09-15T02:13:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-09-15T02:13:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2024-09-15T02:13:21.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:13:21.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:13:21.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:13:21.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2024-09-15T02:13:21.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-09-15T02:13:21.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:13:21.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:13:21.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:13:21.915 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:13:21.915 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:21.915 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:21.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:13:21.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:13:22.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:22.167 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:22.168 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55401/ceph-mon.a.asok 2024-09-15T02:13:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:13:22.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.55401/ceph-mon.a.asok config get fsid 2024-09-15T02:13:22.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:13:22.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:13:22.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55401/ceph-mon.a.asok 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:13:22.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.55401/ceph-mon.a.asok config get mon_host 2024-09-15T02:13:22.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-09-15T02:13:22.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2024-09-15T02:13:22.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:13:22.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:13:22.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:13:22.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2024-09-15T02:13:22.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:13:23.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:13:23.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:13:23.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-09-15T02:13:23.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:13:23.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:13:23.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:13:23.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:23.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:23.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55401/ceph-mon.a.asok 2024-09-15T02:13:23.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-09-15T02:13:23.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.55401/ceph-mon.a.asok log flush 2024-09-15T02:13:23.278 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:13:23.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-09-15T02:13:23.288 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:13:21.940+0000 7f1ecf8c9900 0 load: jerasure load: lrc 2024-09-15T02:13:23.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:13:23.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:23.292 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:13:23.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:13:23.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2024-09-15T02:13:23.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:13:23.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9fef4ce5-1058-42bc-b74c-8af8b1b891b3 2024-09-15T02:13:23.294 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 9fef4ce5-1058-42bc-b74c-8af8b1b891b3 2024-09-15T02:13:23.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9fef4ce5-1058-42bc-b74c-8af8b1b891b3' 2024-09-15T02:13:23.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:13:23.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDQuZmVoAxEhAAnU/3V0Dul4oPluu9NM8WHA== 2024-09-15T02:13:23.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDQuZmVoAxEhAAnU/3V0Dul4oPluu9NM8WHA=="}' 2024-09-15T02:13:23.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9fef4ce5-1058-42bc-b74c-8af8b1b891b3 -i td/test-erasure-code/0/new.json 2024-09-15T02:13:23.578 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:13:23.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2024-09-15T02:13:23.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDQuZmVoAxEhAAnU/3V0Dul4oPluu9NM8WHA== --osd-uuid 9fef4ce5-1058-42bc-b74c-8af8b1b891b3 2024-09-15T02:13:23.612 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:23.611+0000 7f2bffa34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:23.625 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:23.624+0000 7f2bffa34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:23.628 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:23.625+0000 7f2bffa34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:23.628 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:23.625+0000 7f2bffa34440 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-15T02:13:23.628 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:23.625+0000 7f2bffa34440 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-15T02:13:23.628 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:23.625+0000 7f2bffa34440 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-15T02:13:23.665 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:23.664+0000 7f2bffa34440 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-09-15T02:13:29.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-09-15T02:13:29.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:13:29.004 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:13:29.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:13:29.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:13:29.490 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:13:29.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:13:29.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:13:29.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:13:29.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:13:29.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:13:29.512 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:29.511+0000 7ff53540a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:29.514 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:29.513+0000 7ff53540a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:29.515 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:29.514+0000 7ff53540a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:29.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:13:29.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:13:29.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:13:29.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:13:29.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:13:29.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:29.839 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:13:29.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:13:29.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:29.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:30.029 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:30.028+0000 7ff53540a440 -1 Falling back to public interface 2024-09-15T02:13:30.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:31.194 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:13:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:13:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:31.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:32.540 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:32.539+0000 7ff53540a440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:13:32.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:32.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:32.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:13:32.545 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:13:32.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:32.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:32.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:33.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:33.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:33.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:13:33.908 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:13:33.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:33.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:34.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:13:35.367 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:13:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:35.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:35.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:36.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:36.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:36.734 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:13:36.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:13:36.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:36.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:37.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:38.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:38.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:38.216 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:13:38.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:13:38.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:38.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:38.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:39.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:39.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:39.603 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:13:39.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:13:39.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:39.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:39.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:40.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:40.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:40.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:13:40.977 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:13:40.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:40.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:41.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:42.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:42.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:42.383 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:13:42.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:13:42.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:42.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:42.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:43.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:43.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:43.755 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:13:43.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:13:43.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:43.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:44.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:45.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:45.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:45.205 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:13:45.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:13:45.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:45.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:45.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:46.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:46.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:46.667 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:13:46.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:13:46.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:46.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:47.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:48.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:48.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:48.461 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:13:48.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:13:48.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:48.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:48.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:49.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:49.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:49.911 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:13:49.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:13:49.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:49.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:50.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:51.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:51.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:51.609 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:13:51.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:13:51.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:51.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:52.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:53.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:53.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:53.401 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:13:53.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:13:53.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:53.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:54.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:13:55.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:13:55.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:13:55.580 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:13:55.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:13:55.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:13:55.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:13:56.003 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56494,v1:127.0.0.1:6803/56494] [v2:127.0.0.1:6804/56494,v1:127.0.0.1:6805/56494] exists,up 9fef4ce5-1058-42bc-b74c-8af8b1b891b3 2024-09-15T02:13:56.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:13:56.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:13:56.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:13:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a7ad4944-b25d-49c7-ab32-2d81469db9f2 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a7ad4944-b25d-49c7-ab32-2d81469db9f2' 2024-09-15T02:13:56.011 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 a7ad4944-b25d-49c7-ab32-2d81469db9f2 2024-09-15T02:13:56.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:13:56.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkQuZmACktARAAIdFRgSVRI5SAvkJaIIG1RA== 2024-09-15T02:13:56.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkQuZmACktARAAIdFRgSVRI5SAvkJaIIG1RA=="}' 2024-09-15T02:13:56.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a7ad4944-b25d-49c7-ab32-2d81469db9f2 -i td/test-erasure-code/1/new.json 2024-09-15T02:13:56.479 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:13:56.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2024-09-15T02:13:56.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkQuZmACktARAAIdFRgSVRI5SAvkJaIIG1RA== --osd-uuid a7ad4944-b25d-49c7-ab32-2d81469db9f2 2024-09-15T02:13:56.515 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:56.514+0000 7fb36a64d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:56.517 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:56.515+0000 7fb36a64d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:56.517 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:56.516+0000 7fb36a64d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:13:56.518 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:56.517+0000 7fb36a64d440 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-15T02:13:56.518 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:56.517+0000 7fb36a64d440 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-15T02:13:56.518 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:56.517+0000 7fb36a64d440 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-15T02:13:56.596 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:13:56.595+0000 7fb36a64d440 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-09-15T02:14:02.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-09-15T02:14:02.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:14:02.762 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:14:02.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:14:02.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:14:03.236 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:14:03.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:14:03.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:14:03.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:14:03.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:14:03.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:14:03.258 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:03.256+0000 7f7f96615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:03.259 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:03.258+0000 7f7f96615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:03.260 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:03.259+0000 7f7f96615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:03.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:14:03.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:14:03.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:14:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:14:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:14:03.582 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:14:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:14:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:03.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:03.773+0000 7f7f96615440 -1 Falling back to public interface 2024-09-15T02:14:03.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:04.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:04.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:04.943 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:14:04.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:14:04.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:04.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:05.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:06.314 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:06.313+0000 7f7f96615440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:14:06.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:06.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:06.336 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:14:06.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:14:06.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:06.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:06.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:07.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:07.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:07.775 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:14:07.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:14:07.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:07.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:08.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:09.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:09.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:09.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:14:09.214 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:14:09.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:09.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:09.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:10.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:10.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:10.673 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:14:10.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:14:10.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:10.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:11.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:12.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:12.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:12.188 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:14:12.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:14:12.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:12.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:12.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:13.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:13.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:13.567 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:14:13.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:14:13.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:13.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:13.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:14.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:14.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:14.922 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:14:14.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:14:14.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:14.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:15.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:16.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:16.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:16.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:14:16.300 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:14:16.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:16.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:16.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:17.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:17.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:17.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:14:17.712 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:14:17.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:17.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:18.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:19.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:19.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:19.242 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:14:19.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:14:19.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:19.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:20.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:21.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:21.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:21.229 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:14:21.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:14:21.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:21.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:21.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:14:22.964 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:14:22.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:22.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:23.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:24.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:24.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:24.801 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:14:24.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:14:24.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:24.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:25.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:26.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:26.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:26.296 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:14:26.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:14:26.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:26.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:27.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:28.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:28.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:28.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:14:28.065 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:14:28.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:28.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:28.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:29.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:29.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:29.960 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:14:29.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:14:29.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:29.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:14:30.335 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/58248,v1:127.0.0.1:6811/58248] [v2:127.0.0.1:6812/58248,v1:127.0.0.1:6813/58248] exists,up a7ad4944-b25d-49c7-ab32-2d81469db9f2 2024-09-15T02:14:30.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:14:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:14:30.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:14:30.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0709107d-01c6-443d-8a1f-15ec9971eb0b 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0709107d-01c6-443d-8a1f-15ec9971eb0b' 2024-09-15T02:14:30.343 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 0709107d-01c6-443d-8a1f-15ec9971eb0b 2024-09-15T02:14:30.344 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:14:30.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGQ+ZmLUj8FBAAkWyhmqz/BwwadaO/lXSBxw== 2024-09-15T02:14:30.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGQ+ZmLUj8FBAAkWyhmqz/BwwadaO/lXSBxw=="}' 2024-09-15T02:14:30.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0709107d-01c6-443d-8a1f-15ec9971eb0b -i td/test-erasure-code/2/new.json 2024-09-15T02:14:30.872 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:14:30.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2024-09-15T02:14:30.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGQ+ZmLUj8FBAAkWyhmqz/BwwadaO/lXSBxw== --osd-uuid 0709107d-01c6-443d-8a1f-15ec9971eb0b 2024-09-15T02:14:30.907 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:30.906+0000 7f517c51d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:30.931 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:30.930+0000 7f517c51d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:30.932 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:30.931+0000 7f517c51d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:30.933 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:30.931+0000 7f517c51d440 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-15T02:14:30.933 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:30.931+0000 7f517c51d440 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-15T02:14:30.933 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:30.931+0000 7f517c51d440 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-15T02:14:31.014 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:31.013+0000 7f517c51d440 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-09-15T02:14:36.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-09-15T02:14:36.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:14:36.832 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:14:36.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:14:36.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:14:37.312 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:14:37.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:14:37.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:14:37.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:14:37.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:14:37.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:14:37.334 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:37.333+0000 7f6781d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:37.336 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:37.335+0000 7f6781d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:37.337 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:37.336+0000 7f6781d22440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:14:37.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:14:37.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:14:37.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:14:37.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:14:37.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:14:37.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:37.659 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:14:37.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:14:37.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:37.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:37.855 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:37.853+0000 7f6781d22440 -1 Falling back to public interface 2024-09-15T02:14:38.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:39.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:39.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:39.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:14:39.012 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:14:39.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:39.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:39.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:40.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:40.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:40.368 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:14:40.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:14:40.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:40.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:40.384 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:14:40.382+0000 7f6781d22440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:14:40.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:41.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:41.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:41.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:14:41.833 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:14:41.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:41.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:42.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:43.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:43.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:43.383 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:14:43.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:14:43.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:43.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:43.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:44.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:44.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:44.790 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:14:44.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:14:44.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:44.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:45.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:46.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:46.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:46.271 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:14:46.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:14:46.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:46.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:46.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:47.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:47.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:47.661 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:14:47.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:14:47.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:47.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:48.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:49.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:49.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:49.127 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:14:49.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:14:49.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:49.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:49.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:50.599 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:14:50.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:14:50.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:50.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:50.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:51.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:51.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:51.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:14:51.981 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:14:51.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:51.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:52.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:53.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:53.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:53.704 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:14:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:14:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:54.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:55.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:55.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:55.457 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:14:55.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:14:55.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:55.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:56.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:57.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:57.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:57.376 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:14:57.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:14:57.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:57.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:14:58.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:14:59.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:14:59.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:14:59.178 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:14:59.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:14:59.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:14:59.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:15:00.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:01.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:01.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:01.317 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:15:01.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:15:01.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:01.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:15:02.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:03.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:03.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:03.012 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:15:03.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:15:03.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:03.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:15:03.493 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/60003,v1:127.0.0.1:6819/60003] [v2:127.0.0.1:6820/60003,v1:127.0.0.1:6821/60003] exists,up 0709107d-01c6-443d-8a1f-15ec9971eb0b 2024-09-15T02:15:03.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:15:03.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:15:03.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:15:03.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:15:03.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-09-15T02:15:03.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:15:03.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:15:03.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c2e2d5f-3a7f-4bd6-853b-66ed7d22fdd5 2024-09-15T02:15:03.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5c2e2d5f-3a7f-4bd6-853b-66ed7d22fdd5' 2024-09-15T02:15:03.501 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 5c2e2d5f-3a7f-4bd6-853b-66ed7d22fdd5 2024-09-15T02:15:03.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:15:03.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnQ+ZmO/hjHhAA3kP/xImuwiEcEMhl5evDTQ== 2024-09-15T02:15:03.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnQ+ZmO/hjHhAA3kP/xImuwiEcEMhl5evDTQ=="}' 2024-09-15T02:15:03.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c2e2d5f-3a7f-4bd6-853b-66ed7d22fdd5 -i td/test-erasure-code/3/new.json 2024-09-15T02:15:04.099 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:15:04.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2024-09-15T02:15:04.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnQ+ZmO/hjHhAA3kP/xImuwiEcEMhl5evDTQ== --osd-uuid 5c2e2d5f-3a7f-4bd6-853b-66ed7d22fdd5 2024-09-15T02:15:04.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:04.133+0000 7f88eb31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:04.136 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:04.135+0000 7f88eb31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:04.137 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:04.136+0000 7f88eb31d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:04.138 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:04.136+0000 7f88eb31d440 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-15T02:15:04.138 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:04.136+0000 7f88eb31d440 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-15T02:15:04.138 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:04.136+0000 7f88eb31d440 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-15T02:15:04.181 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:04.180+0000 7f88eb31d440 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-09-15T02:15:10.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-09-15T02:15:10.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:15:10.138 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:15:10.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:15:10.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:15:10.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:15:10.619 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:15:10.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:15:10.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:15:10.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:15:10.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:15:10.640 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:10.639+0000 7f198451f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:10.642 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:10.641+0000 7f198451f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:10.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:10.642+0000 7f198451f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:10.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:15:10.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:15:10.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:15:10.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:15:10.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:15:10.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:10.970 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:15:10.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:15:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:11.159 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:11.158+0000 7f198451f440 -1 Falling back to public interface 2024-09-15T02:15:11.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:12.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:12.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:12.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:15:12.330 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:15:12.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:12.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:12.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:13.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:13.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:13.685 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:15:13.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:15:13.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:13.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:13.727 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:13.726+0000 7f198451f440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:15:14.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:15.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:15.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:15.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:15:15.062 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:15:15.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:15.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:15.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:16.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:16.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:16.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:15:16.564 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:15:16.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:16.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:16.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:17.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:17.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:17.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:15:17.974 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:15:17.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:17.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:18.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:19.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:19.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:19.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:15:19.338 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:15:19.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:19.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:19.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:20.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:20.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:20.828 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:15:20.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:15:20.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:20.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:21.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:22.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:22.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:22.218 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:15:22.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:15:22.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:22.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:22.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:23.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:23.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:23.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:15:23.674 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:15:23.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:23.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:24.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:25.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:25.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:25.117 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:15:25.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:15:25.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:25.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:25.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:26.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:26.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:26.535 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:15:26.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:15:26.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:26.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:27.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:28.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:28.466 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:15:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:15:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:29.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:30.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:30.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:30.319 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:15:30.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:15:30.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:30.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:31.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:32.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:32.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:32.180 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:15:32.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:15:32.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:32.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:32.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:33.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:33.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:33.803 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:15:33.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:15:33.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:33.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:34.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:35.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:35.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:35.250 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:15:35.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:15:35.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:35.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:36.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:37.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:37.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:37.118 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:15:37.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:15:37.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:37.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:15:37.534 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/61732,v1:127.0.0.1:6827/61732] [v2:127.0.0.1:6828/61732,v1:127.0.0.1:6829/61732] exists,up 5c2e2d5f-3a7f-4bd6-853b-66ed7d22fdd5 2024-09-15T02:15:37.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:15:37.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:15:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:15:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2024-09-15T02:15:37.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:15:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8aacd2b3-8f69-41fb-a685-3e36534a9dc2 2024-09-15T02:15:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 8aacd2b3-8f69-41fb-a685-3e36534a9dc2' 2024-09-15T02:15:37.542 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 8aacd2b3-8f69-41fb-a685-3e36534a9dc2 2024-09-15T02:15:37.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:15:37.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJQ+ZmKRncIBAAVflN/NERCVDb/fKn7YH0uw== 2024-09-15T02:15:37.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJQ+ZmKRncIBAAVflN/NERCVDb/fKn7YH0uw=="}' 2024-09-15T02:15:37.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8aacd2b3-8f69-41fb-a685-3e36534a9dc2 -i td/test-erasure-code/4/new.json 2024-09-15T02:15:38.095 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:15:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2024-09-15T02:15:38.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJQ+ZmKRncIBAAVflN/NERCVDb/fKn7YH0uw== --osd-uuid 8aacd2b3-8f69-41fb-a685-3e36534a9dc2 2024-09-15T02:15:38.130 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:38.128+0000 7f4dd2f1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:38.132 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:38.130+0000 7f4dd2f1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:38.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:38.131+0000 7f4dd2f1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:38.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:38.131+0000 7f4dd2f1e440 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-15T02:15:38.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:38.131+0000 7f4dd2f1e440 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-15T02:15:38.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:38.131+0000 7f4dd2f1e440 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-15T02:15:38.186 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:38.184+0000 7f4dd2f1e440 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-09-15T02:15:44.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-09-15T02:15:44.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:15:44.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:15:44.257 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:15:44.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:15:44.661 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:15:44.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:15:44.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:15:44.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:15:44.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:15:44.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:15:44.684 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:44.683+0000 7fa208017440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:44.686 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:44.684+0000 7fa208017440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:44.687 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:44.685+0000 7fa208017440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:15:45.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:15:45.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:15:45.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:15:45.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:15:45.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:15:45.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:45.012 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:15:45.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:15:45.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:45.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:45.203 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:45.202+0000 7fa208017440 -1 Falling back to public interface 2024-09-15T02:15:45.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:46.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:46.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:46.361 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:15:46.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:15:46.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:46.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:46.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:15:47.716 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:15:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:47.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:47.783 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:15:47.782+0000 7fa208017440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:15:48.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:49.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:49.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:49.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:15:49.151 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:15:49.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:49.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:49.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:50.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:50.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:50.616 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:15:50.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:15:50.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:50.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:50.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:51.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:51.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:51.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:15:51.997 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:15:51.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:51.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:52.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:53.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:53.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:53.404 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:15:53.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:15:53.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:53.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:53.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:54.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:54.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:54.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:15:54.880 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:15:54.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:54.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:55.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:56.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:56.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:56.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:15:56.260 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:15:56.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:56.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:56.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:57.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:57.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:57.726 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:15:57.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:15:57.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:57.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:58.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:15:59.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:15:59.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:15:59.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:15:59.087 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:15:59.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:15:59.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:15:59.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:00.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:00.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:00.482 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:16:00.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:16:00.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:00.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:16:01.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:02.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:02.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:02.483 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:16:02.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:16:02.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:02.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:16:03.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:04.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:04.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:04.199 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:16:04.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:16:04.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:04.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:16:04.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:05.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:05.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:05.864 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:16:05.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:16:05.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:05.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:16:06.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:07.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:07.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:07.869 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:16:07.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:16:07.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:07.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:16:08.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:09.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:09.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:09.621 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:16:09.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:16:09.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:09.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:16:10.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:11.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:11.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:11.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:16:11.217 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:16:11.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:11.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:16:11.601 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/63486,v1:127.0.0.1:6835/63486] [v2:127.0.0.1:6836/63486,v1:127.0.0.1:6837/63486] exists,up 8aacd2b3-8f69-41fb-a685-3e36534a9dc2 2024-09-15T02:16:11.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:16:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:16:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:16:11.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:16:11.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2024-09-15T02:16:11.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:16:11.609 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 ad685a9c-f09a-44c7-9f9c-c61d75840abb 2024-09-15T02:16:11.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ad685a9c-f09a-44c7-9f9c-c61d75840abb 2024-09-15T02:16:11.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ad685a9c-f09a-44c7-9f9c-c61d75840abb' 2024-09-15T02:16:11.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:16:11.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrQ+ZmFjDaJBAAzf3PaWxa8sSEhkB8iVGA5g== 2024-09-15T02:16:11.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrQ+ZmFjDaJBAAzf3PaWxa8sSEhkB8iVGA5g=="}' 2024-09-15T02:16:11.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ad685a9c-f09a-44c7-9f9c-c61d75840abb -i td/test-erasure-code/5/new.json 2024-09-15T02:16:12.106 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:16:12.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2024-09-15T02:16:12.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrQ+ZmFjDaJBAAzf3PaWxa8sSEhkB8iVGA5g== --osd-uuid ad685a9c-f09a-44c7-9f9c-c61d75840abb 2024-09-15T02:16:12.141 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:12.139+0000 7fbc8de0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:12.142 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:12.141+0000 7fbc8de0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:12.144 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:12.142+0000 7fbc8de0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:12.144 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:12.142+0000 7fbc8de0d440 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-15T02:16:12.144 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:12.142+0000 7fbc8de0d440 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-15T02:16:12.144 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:12.142+0000 7fbc8de0d440 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-15T02:16:12.230 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:12.228+0000 7fbc8de0d440 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-09-15T02:16:17.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-09-15T02:16:17.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:16:17.797 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-15T02:16:17.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T02:16:17.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:16:18.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T02:16:18.278 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-15T02:16:18.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:16:18.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:16:18.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:16:18.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:16:18.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:18.299+0000 7f39cd48b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:18.302 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:18.301+0000 7f39cd48b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:18.303 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:18.302+0000 7f39cd48b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:18.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T02:16:18.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:16:18.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T02:16:18.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:16:18.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:16:18.631 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:16:18.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:18.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:16:18.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:18.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:18.816 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:18.815+0000 7f39cd48b440 -1 Falling back to public interface 2024-09-15T02:16:18.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:19.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:19.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:19.983 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:16:19.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:16:19.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:19.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:20.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:21.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:21.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:21.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:16:21.336 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:16:21.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:21.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:21.370 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:21.369+0000 7f39cd48b440 -1 osd.5 0 log_to_monitors true 2024-09-15T02:16:21.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:22.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:22.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:22.701 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:16:22.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:16:22.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:22.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:23.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:24.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:24.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:24.307 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:16:24.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:16:24.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:24.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:24.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:25.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:25.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:25.694 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:16:25.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:16:25.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:25.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:26.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:27.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:27.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:27.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:16:27.059 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:16:27.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:27.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:27.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:28.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:28.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:28.573 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:16:28.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:16:28.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:28.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:28.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:29.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:29.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:29.966 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:16:29.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:16:29.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:29.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:31.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:31.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:31.341 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:16:31.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:16:31.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:31.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:31.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:32.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:32.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:32.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:16:32.704 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:16:32.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:32.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:33.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:34.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:34.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:34.095 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:16:34.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:16:34.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:34.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:34.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:35.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:35.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:35.809 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:16:35.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:16:35.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:35.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:37.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:37.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:37.319 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:16:37.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:16:37.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:37.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:38.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:39.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:39.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:39.375 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:16:39.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:16:39.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:39.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:40.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:41.112 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:16:41.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:16:41.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:41.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:41.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:42.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:42.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:42.723 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:16:42.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:16:42.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:42.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:43.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:44.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:44.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:44.567 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:16:44.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:16:44.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:44.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:16:44.946 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/65241,v1:127.0.0.1:6843/65241] [v2:127.0.0.1:6844/65241,v1:127.0.0.1:6845/65241] exists,up ad685a9c-f09a-44c7-9f9c-c61d75840abb 2024-09-15T02:16:44.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:16:44.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:16:44.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:16:44.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:16:44.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-15T02:16:44.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:16:44.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:16:44.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:16:44.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:16:44.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2024-09-15T02:16:44.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:16:44.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6787b80-2616-456a-a3e2-25d3b0e67d39 2024-09-15T02:16:44.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 c6787b80-2616-456a-a3e2-25d3b0e67d39' 2024-09-15T02:16:44.954 INFO:tasks.workunit.client.0.smithi130.stdout:add osd6 c6787b80-2616-456a-a3e2-25d3b0e67d39 2024-09-15T02:16:44.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:16:44.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMQ+Zm6JhsORAAvLYrRWZzqTD9UjP2S+xEog== 2024-09-15T02:16:44.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMQ+Zm6JhsORAAvLYrRWZzqTD9UjP2S+xEog=="}' 2024-09-15T02:16:44.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6787b80-2616-456a-a3e2-25d3b0e67d39 -i td/test-erasure-code/6/new.json 2024-09-15T02:16:45.604 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:16:45.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2024-09-15T02:16:45.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMQ+Zm6JhsORAAvLYrRWZzqTD9UjP2S+xEog== --osd-uuid c6787b80-2616-456a-a3e2-25d3b0e67d39 2024-09-15T02:16:45.640 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:45.638+0000 7f892d31f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:45.641 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:45.640+0000 7f892d31f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:45.642 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:45.641+0000 7f892d31f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:45.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:45.641+0000 7f892d31f440 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-15T02:16:45.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:45.641+0000 7f892d31f440 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-15T02:16:45.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:45.642+0000 7f892d31f440 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-15T02:16:45.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:45.739+0000 7f892d31f440 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-09-15T02:16:51.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-09-15T02:16:51.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:16:51.551 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd6 key to auth repository 2024-09-15T02:16:51.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-15T02:16:51.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:16:52.037 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.6 2024-09-15T02:16:52.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-15T02:16:52.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:16:52.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:16:52.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:16:52.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:16:52.059 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:52.058+0000 7f6795129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:52.061 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:52.060+0000 7f6795129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:52.062 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:52.061+0000 7f6795129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:16:52.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-15T02:16:52.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:16:52.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-15T02:16:52.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:16:52.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:16:52.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:52.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:16:52.386 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:16:52.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:52.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:16:52.579 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:52.578+0000 7f6795129440 -1 Falling back to public interface 2024-09-15T02:16:52.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:53.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:53.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:53.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:16:53.737 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:16:53.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:53.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:16:54.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:55.047 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:16:55.046+0000 7f6795129440 -1 osd.6 0 log_to_monitors true 2024-09-15T02:16:55.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:55.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:55.085 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:16:55.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:16:55.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:55.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:16:55.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:56.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:56.591 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:16:56.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:56.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:16:56.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:56.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:16:57.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:58.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:58.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:58.132 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:16:58.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:16:58.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:58.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:16:58.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:16:59.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:16:59.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:16:59.540 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:16:59.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:16:59.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:16:59.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:00.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:01.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:01.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:01.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:17:01.030 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:17:01.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:01.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:01.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:02.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:02.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:02.499 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:17:02.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:17:02.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:02.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:02.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:03.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:03.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:03.977 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:17:03.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:17:03.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:03.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:04.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:05.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:05.391 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:17:05.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:17:05.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:05.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:05.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:06.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:06.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:06.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:17:06.909 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:17:06.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:06.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:07.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:08.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:08.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:08.576 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:17:08.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:17:08.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:08.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:09.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:10.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:10.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:10.196 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:17:10.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:17:10.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:10.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:10.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:11.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:11.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:11.926 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:17:11.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:17:11.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:11.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:12.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:13.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:13.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:13.685 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:17:13.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:17:13.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:13.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:14.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:15.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:15.482 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:17:15.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:17:15.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:15.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:16.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:17.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:17.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:17.200 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:17:17.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:17:17.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:17.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:17.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:18.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:18.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:18.609 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:17:18.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:17:18.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:18.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:17:18.959 INFO:tasks.workunit.client.0.smithi130.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/66995,v1:127.0.0.1:6851/66995] [v2:127.0.0.1:6852/66995,v1:127.0.0.1:6853/66995] exists,up c6787b80-2616-456a-a3e2-25d3b0e67d39 2024-09-15T02:17:18.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:17:18.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:17:18.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:17:18.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-09-15T02:17:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:17:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f430ac69-7926-4d99-b03a-b89fce97ac52 2024-09-15T02:17:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 f430ac69-7926-4d99-b03a-b89fce97ac52' 2024-09-15T02:17:18.967 INFO:tasks.workunit.client.0.smithi130.stdout:add osd7 f430ac69-7926-4d99-b03a-b89fce97ac52 2024-09-15T02:17:18.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:17:18.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuQ+ZmpjkzOhAAKN4VJCV7y7BiBHxwaM3IQw== 2024-09-15T02:17:18.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuQ+ZmpjkzOhAAKN4VJCV7y7BiBHxwaM3IQw=="}' 2024-09-15T02:17:18.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f430ac69-7926-4d99-b03a-b89fce97ac52 -i td/test-erasure-code/7/new.json 2024-09-15T02:17:19.503 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:17:19.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2024-09-15T02:17:19.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCuQ+ZmpjkzOhAAKN4VJCV7y7BiBHxwaM3IQw== --osd-uuid f430ac69-7926-4d99-b03a-b89fce97ac52 2024-09-15T02:17:19.539 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:19.537+0000 7fb6ed615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:19.640 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:19.638+0000 7fb6ed615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:19.641 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:19.639+0000 7fb6ed615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:19.641 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:19.640+0000 7fb6ed615440 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-15T02:17:19.641 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:19.640+0000 7fb6ed615440 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-15T02:17:19.642 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:19.640+0000 7fb6ed615440 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-15T02:17:19.679 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:19.679+0000 7fb6ed615440 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-09-15T02:17:25.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-09-15T02:17:25.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:17:25.519 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd7 key to auth repository 2024-09-15T02:17:25.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-15T02:17:25.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:17:26.003 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.7 2024-09-15T02:17:26.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-15T02:17:26.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:17:26.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:17:26.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:17:26.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:17:26.025 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:26.024+0000 7f754ad88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:26.027 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:26.026+0000 7f754ad88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:26.028 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:26.027+0000 7f754ad88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:26.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-15T02:17:26.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:17:26.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-15T02:17:26.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:17:26.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:17:26.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:26.352 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:17:26.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:17:26.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:26.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:26.552 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:26.551+0000 7f754ad88440 -1 Falling back to public interface 2024-09-15T02:17:26.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:27.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:27.708 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:17:27.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:17:27.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:27.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:28.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:29.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:29.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:29.059 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:17:29.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:17:29.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:29.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:29.079 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:29.077+0000 7f754ad88440 -1 osd.7 0 log_to_monitors true 2024-09-15T02:17:29.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:30.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:30.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:30.549 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:17:30.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:17:30.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:30.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:31.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:32.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:32.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:32.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:17:32.081 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:17:32.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:32.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:32.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:33.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:33.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:33.458 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:17:33.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:17:33.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:33.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:33.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:34.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:34.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:34.846 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:17:34.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:17:34.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:34.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:35.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:36.225 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:17:36.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:17:36.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:36.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:36.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:37.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:37.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:37.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:17:37.584 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:17:37.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:37.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:38.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:39.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:39.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:39.089 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:17:39.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:17:39.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:39.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:39.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:40.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:40.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:40.572 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:17:40.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:17:40.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:40.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:42.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:42.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:42.053 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:17:42.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:17:42.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:42.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:43.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:43.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:43.758 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:17:43.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:17:43.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:43.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:45.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:45.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:45.354 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:17:45.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:17:45.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:45.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:46.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:47.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:47.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:47.578 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:17:47.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:17:47.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:47.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:48.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:49.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:49.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:49.355 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:17:49.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:17:49.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:49.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:50.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:51.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:51.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:51.068 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:17:51.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:17:51.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:51.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:17:52.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:17:52.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:17:52.656 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:17:52.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:17:52.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:17:52.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:17:53.056 INFO:tasks.workunit.client.0.smithi130.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/68749,v1:127.0.0.1:6859/68749] [v2:127.0.0.1:6860/68749,v1:127.0.0.1:6861/68749] exists,up f430ac69-7926-4d99-b03a-b89fce97ac52 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:17:53.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:17:53.061 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:17:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:17:53.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2024-09-15T02:17:53.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:17:53.063 INFO:tasks.workunit.client.0.smithi130.stdout:add osd8 ff0b3f9a-c1f8-4086-b190-08c1a1ae44e1 2024-09-15T02:17:53.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff0b3f9a-c1f8-4086-b190-08c1a1ae44e1 2024-09-15T02:17:53.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 ff0b3f9a-c1f8-4086-b190-08c1a1ae44e1' 2024-09-15T02:17:53.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:17:53.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRQ+ZmWfdWBBAAYPJkJV0Gc/DWwkgk8trhqg== 2024-09-15T02:17:53.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRQ+ZmWfdWBBAAYPJkJV0Gc/DWwkgk8trhqg=="}' 2024-09-15T02:17:53.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff0b3f9a-c1f8-4086-b190-08c1a1ae44e1 -i td/test-erasure-code/8/new.json 2024-09-15T02:17:53.650 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:17:53.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2024-09-15T02:17:53.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRQ+ZmWfdWBBAAYPJkJV0Gc/DWwkgk8trhqg== --osd-uuid ff0b3f9a-c1f8-4086-b190-08c1a1ae44e1 2024-09-15T02:17:53.686 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:53.684+0000 7fa55bc65440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:53.691 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:53.690+0000 7fa55bc65440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:53.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:53.691+0000 7fa55bc65440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:53.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:53.691+0000 7fa55bc65440 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-15T02:17:53.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:53.691+0000 7fa55bc65440 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-15T02:17:53.693 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:53.691+0000 7fa55bc65440 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-15T02:17:53.782 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:53.781+0000 7fa55bc65440 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-09-15T02:17:59.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-09-15T02:17:59.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:17:59.362 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd8 key to auth repository 2024-09-15T02:17:59.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-09-15T02:17:59.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:17:59.849 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.8 2024-09-15T02:17:59.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-09-15T02:17:59.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:17:59.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:17:59.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:17:59.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:17:59.871 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:59.869+0000 7fc39f808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:59.872 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:59.871+0000 7fc39f808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:17:59.873 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:17:59.872+0000 7fc39f808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-09-15T02:18:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:18:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-15T02:18:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:18:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:18:00.202 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:18:00.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:00.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:18:00.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:00.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:00.388 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:00.387+0000 7fc39f808440 -1 Falling back to public interface 2024-09-15T02:18:00.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:01.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:01.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:01.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:18:01.555 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:18:01.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:01.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:01.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:02.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:02.921 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:18:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:18:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:02.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:03.014 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:03.013+0000 7fc39f808440 -1 osd.8 0 log_to_monitors true 2024-09-15T02:18:03.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:04.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:04.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:04.331 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:18:04.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:18:04.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:04.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:04.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:05.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:05.793 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:18:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:18:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:05.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:06.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:07.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:07.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:07.229 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:18:07.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:18:07.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:07.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:07.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:08.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:08.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:08.621 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:18:08.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:18:08.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:08.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:09.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:10.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:10.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:10.102 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:18:10.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:18:10.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:10.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:10.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:11.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:11.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:11.484 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:18:11.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:18:11.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:11.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:11.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:12.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:12.983 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:18:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:18:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:12.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:13.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:14.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:14.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:14.348 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:18:14.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:18:14.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:14.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:14.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:15.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:15.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:15.851 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:18:15.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:18:15.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:15.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:16.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:17.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:17.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:17.329 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:18:17.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:18:17.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:17.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:17.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:18.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:18.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:18.934 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:18:18.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:18:18.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:18.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:19.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:20.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:20.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:20.820 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:18:20.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:18:20.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:20.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:21.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:22.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:22.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:22.179 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:18:22.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:18:22.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:22.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:22.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:23.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:23.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:23.934 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:18:23.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:18:23.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:23.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:24.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:25.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:25.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:25.849 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:18:25.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:18:25.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:25.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:18:26.240 INFO:tasks.workunit.client.0.smithi130.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/70504,v1:127.0.0.1:6867/70504] [v2:127.0.0.1:6868/70504,v1:127.0.0.1:6869/70504] exists,up ff0b3f9a-c1f8-4086-b190-08c1a1ae44e1 2024-09-15T02:18:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:18:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:18:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:18:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:18:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-09-15T02:18:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:18:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:18:26.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:18:26.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30b27c15-d1a2-48cc-9f0c-967dcae995fe 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 30b27c15-d1a2-48cc-9f0c-967dcae995fe' 2024-09-15T02:18:26.247 INFO:tasks.workunit.client.0.smithi130.stdout:add osd9 30b27c15-d1a2-48cc-9f0c-967dcae995fe 2024-09-15T02:18:26.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:18:26.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyQ+Zm/CpVDxAAxt4toyMzhOcCbTn3cxlRZg== 2024-09-15T02:18:26.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyQ+Zm/CpVDxAAxt4toyMzhOcCbTn3cxlRZg=="}' 2024-09-15T02:18:26.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30b27c15-d1a2-48cc-9f0c-967dcae995fe -i td/test-erasure-code/9/new.json 2024-09-15T02:18:26.754 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:18:26.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2024-09-15T02:18:26.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyQ+Zm/CpVDxAAxt4toyMzhOcCbTn3cxlRZg== --osd-uuid 30b27c15-d1a2-48cc-9f0c-967dcae995fe 2024-09-15T02:18:26.790 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:26.788+0000 7f963d51f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:26.792 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:26.790+0000 7f963d51f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:26.793 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:26.791+0000 7f963d51f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:26.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:26.791+0000 7f963d51f440 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-15T02:18:26.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:26.791+0000 7f963d51f440 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-15T02:18:26.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:26.791+0000 7f963d51f440 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-15T02:18:26.912 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:26.910+0000 7f963d51f440 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-09-15T02:18:32.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-09-15T02:18:32.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:18:32.771 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd9 key to auth repository 2024-09-15T02:18:32.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-09-15T02:18:32.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:18:33.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-09-15T02:18:33.256 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.9 2024-09-15T02:18:33.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:18:33.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:18:33.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:18:33.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:18:33.278 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:33.276+0000 7fe8e9b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:33.280 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:33.278+0000 7fe8e9b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:33.281 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:33.279+0000 7fe8e9b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:33.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-09-15T02:18:33.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:18:33.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-15T02:18:33.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:18:33.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:18:33.606 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:18:33.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:33.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:18:33.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:33.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:33.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:33.800+0000 7fe8e9b27440 -1 Falling back to public interface 2024-09-15T02:18:33.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:34.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:34.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:34.957 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:18:34.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:18:34.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:35.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:36.269 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:36.268+0000 7fe8e9b27440 -1 osd.9 0 log_to_monitors true 2024-09-15T02:18:36.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:36.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:36.316 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:18:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:18:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:36.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:36.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:37.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:37.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:37.819 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:18:37.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:18:37.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:37.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:38.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:39.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:39.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:39.215 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:18:39.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:18:39.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:39.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:39.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:40.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:40.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:40.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:18:40.685 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:18:40.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:40.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:41.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:42.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:42.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:42.201 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:18:42.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:18:42.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:42.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:42.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:43.591 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:18:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:18:43.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:43.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:44.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:45.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:45.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:45.008 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:18:45.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:18:45.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:45.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:45.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:46.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:46.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:46.540 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:18:46.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:18:46.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:46.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:47.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:48.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:48.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:48.079 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:18:48.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:18:48.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:48.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:48.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:49.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:49.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:49.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:18:49.742 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:18:49.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:49.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:51.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:51.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:51.504 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:18:51.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:18:51.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:51.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:52.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:53.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:53.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:53.282 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:18:53.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:18:53.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:53.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:54.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:55.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:55.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:55.296 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:18:55.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:18:55.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:55.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:55.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:56.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:56.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:56.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:18:56.961 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:18:56.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:56.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:57.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:18:58.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:18:58.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:18:58.873 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:18:58.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:18:58.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:18:58.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:18:59.233 INFO:tasks.workunit.client.0.smithi130.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/72258,v1:127.0.0.1:6875/72258] [v2:127.0.0.1:6876/72258,v1:127.0.0.1:6877/72258] exists,up 30b27c15-d1a2-48cc-9f0c-967dcae995fe 2024-09-15T02:18:59.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:18:59.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:18:59.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:18:59.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-15T02:18:59.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-09-15T02:18:59.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-15T02:18:59.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:18:59.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2024-09-15T02:18:59.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:18:59.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:18:59.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2024-09-15T02:18:59.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:18:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3df42b5d-6b27-4ed0-b21b-830385f70861 2024-09-15T02:18:59.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 3df42b5d-6b27-4ed0-b21b-830385f70861' 2024-09-15T02:18:59.246 INFO:tasks.workunit.client.0.smithi130.stdout:add osd10 3df42b5d-6b27-4ed0-b21b-830385f70861 2024-09-15T02:18:59.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:18:59.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATROZm+I/nDhAADk61EedWGUXOhJVIy2diIQ== 2024-09-15T02:18:59.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATROZm+I/nDhAADk61EedWGUXOhJVIy2diIQ=="}' 2024-09-15T02:18:59.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3df42b5d-6b27-4ed0-b21b-830385f70861 -i td/test-erasure-code/10/new.json 2024-09-15T02:18:59.743 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:18:59.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2024-09-15T02:18:59.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATROZm+I/nDhAADk61EedWGUXOhJVIy2diIQ== --osd-uuid 3df42b5d-6b27-4ed0-b21b-830385f70861 2024-09-15T02:18:59.779 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:59.777+0000 7fd36020a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:59.801 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:59.800+0000 7fd36020a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:59.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:59.801+0000 7fd36020a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:18:59.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:59.801+0000 7fd36020a440 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-15T02:18:59.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:59.801+0000 7fd36020a440 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-15T02:18:59.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:59.801+0000 7fd36020a440 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-15T02:18:59.892 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:18:59.891+0000 7fd36020a440 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-09-15T02:19:05.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-09-15T02:19:05.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:19:05.727 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd10 key to auth repository 2024-09-15T02:19:05.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2024-09-15T02:19:05.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:19:06.209 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.10 2024-09-15T02:19:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2024-09-15T02:19:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:19:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:19:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:19:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:19:06.232 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:19:06.230+0000 7f5bd4e10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:19:06.253 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:19:06.252+0000 7f5bd4e10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:19:06.254 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:19:06.253+0000 7f5bd4e10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:19:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2024-09-15T02:19:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:19:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2024-09-15T02:19:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:19:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:19:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:06.596 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:19:06.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:19:06.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:06.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:06.781 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:19:06.780+0000 7f5bd4e10440 -1 Falling back to public interface 2024-09-15T02:19:06.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:07.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:07.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:07.975 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:19:07.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:19:07.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:07.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:08.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:09.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:09.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:09.335 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:19:09.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:19:09.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:09.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:09.391 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:19:09.389+0000 7f5bd4e10440 -1 osd.10 0 log_to_monitors true 2024-09-15T02:19:09.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:10.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:10.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:10.718 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:19:10.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:19:10.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:10.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:11.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:12.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:12.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:12.137 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:19:12.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:19:12.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:12.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:12.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:13.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:13.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:13.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:19:13.592 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:19:13.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:13.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:14.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:15.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:15.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:15.090 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:19:15.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:19:15.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:15.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:15.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:16.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:16.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:16.624 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:19:16.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:19:16.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:16.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:16.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:17.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:17.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:17.987 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:19:17.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:19:17.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:17.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:18.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:19.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:19.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:19.486 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:19:19.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:19:19.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:19.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:19.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:20.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:20.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:20.985 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:19:20.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:19:20.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:20.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:21.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:22.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:22.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:22.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:19:22.460 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:19:22.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:22.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:23.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:24.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:24.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:24.187 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:19:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:19:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:24.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:25.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:25.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:25.908 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:19:25.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:19:25.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:25.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:26.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:27.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:27.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:27.771 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:19:27.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:19:27.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:27.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:28.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:29.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:29.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:29.409 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:19:29.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:19:29.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:29.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:30.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:31.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:31.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:31.553 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:19:31.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:19:31.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:31.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:32.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:33.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:33.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:33.357 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:19:33.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:19:33.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:33.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:33.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:19:34.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:19:34.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:19:34.719 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:19:34.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:19:34.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:19:34.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-15T02:19:35.098 INFO:tasks.workunit.client.0.smithi130.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/73988,v1:127.0.0.1:6883/73988] [v2:127.0.0.1:6884/73988,v1:127.0.0.1:6885/73988] exists,up 3df42b5d-6b27-4ed0-b21b-830385f70861 2024-09-15T02:19:35.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:19:35.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:19:35.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:19:35.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-09-15T02:19:35.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T02:19:35.442 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' does not exist 2024-09-15T02:19:35.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T02:19:35.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:19:36.034 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:19:36.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:19:37.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T02:19:40.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-09-15T02:19:40.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:19:40.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:19:40.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:19:40.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:19:40.799 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:19:40.799 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:19:40.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:19:40.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:19:40.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:19:40.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:19:40.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:19:40.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:19:40.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:19:40.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:19:40.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:19:41.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:19:41.268 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:19:41.268 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:19:41.268 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:41.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:19:41.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836552 2024-09-15T02:19:41.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836552 2024-09-15T02:19:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552' 2024-09-15T02:19:41.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:41.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:19:41.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673031 2024-09-15T02:19:41.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673031 2024-09-15T02:19:41.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031' 2024-09-15T02:19:41.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:41.535 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:19:41.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509503 2024-09-15T02:19:41.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509503 2024-09-15T02:19:41.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503' 2024-09-15T02:19:41.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:41.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:19:41.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345978 2024-09-15T02:19:41.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345978 2024-09-15T02:19:41.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978' 2024-09-15T02:19:41.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:41.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:19:41.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182451 2024-09-15T02:19:41.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182451 2024-09-15T02:19:41.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978 4-107374182451' 2024-09-15T02:19:41.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:41.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:19:41.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018922 2024-09-15T02:19:41.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018922 2024-09-15T02:19:41.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978 4-107374182451 5-128849018922' 2024-09-15T02:19:41.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:41.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:19:42.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855392 2024-09-15T02:19:42.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855392 2024-09-15T02:19:42.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978 4-107374182451 5-128849018922 6-150323855392' 2024-09-15T02:19:42.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:42.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:19:42.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691868 2024-09-15T02:19:42.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691868 2024-09-15T02:19:42.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978 4-107374182451 5-128849018922 6-150323855392 7-171798691868' 2024-09-15T02:19:42.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:42.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:19:42.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528344 2024-09-15T02:19:42.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528344 2024-09-15T02:19:42.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978 4-107374182451 5-128849018922 6-150323855392 7-171798691868 8-193273528344' 2024-09-15T02:19:42.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:42.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:19:42.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364813 2024-09-15T02:19:42.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364813 2024-09-15T02:19:42.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978 4-107374182451 5-128849018922 6-150323855392 7-171798691868 8-193273528344 9-214748364813' 2024-09-15T02:19:42.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:42.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:19:42.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201290 2024-09-15T02:19:42.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201290 2024-09-15T02:19:42.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836552 1-42949673031 2-64424509503 3-85899345978 4-107374182451 5-128849018922 6-150323855392 7-171798691868 8-193273528344 9-214748364813 10-236223201290' 2024-09-15T02:19:42.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:42.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836552 2024-09-15T02:19:42.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:42.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:19:42.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836552 2024-09-15T02:19:42.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:42.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836552 2024-09-15T02:19:42.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-09-15T02:19:42.479 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836552 2024-09-15T02:19:42.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:19:42.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836549 -lt 21474836552 2024-09-15T02:19:42.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:19:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:19:43.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:19:44.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836552 -lt 21474836552 2024-09-15T02:19:44.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:44.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673031 2024-09-15T02:19:44.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:44.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:19:44.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673031 2024-09-15T02:19:44.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:44.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673031 2024-09-15T02:19:44.263 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673031 2024-09-15T02:19:44.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-09-15T02:19:44.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:19:44.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673031 -lt 42949673031 2024-09-15T02:19:44.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:44.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509503 2024-09-15T02:19:44.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:44.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:19:44.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509503 2024-09-15T02:19:44.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:44.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509503 2024-09-15T02:19:44.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-09-15T02:19:44.617 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509503 2024-09-15T02:19:44.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:19:44.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509504 -lt 64424509503 2024-09-15T02:19:44.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:44.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345978 2024-09-15T02:19:44.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:44.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:19:44.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345978 2024-09-15T02:19:44.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:44.993 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345978 2024-09-15T02:19:44.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345978 2024-09-15T02:19:44.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345978' 2024-09-15T02:19:44.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:19:45.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345979 -lt 85899345978 2024-09-15T02:19:45.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:45.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182451 2024-09-15T02:19:45.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:45.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:19:45.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182451 2024-09-15T02:19:45.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:45.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182451 2024-09-15T02:19:45.347 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182451 2024-09-15T02:19:45.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-09-15T02:19:45.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:19:45.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182451 -lt 107374182451 2024-09-15T02:19:45.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:45.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018922 2024-09-15T02:19:45.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:45.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:19:45.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018922 2024-09-15T02:19:45.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:45.740 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018922 2024-09-15T02:19:45.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018922 2024-09-15T02:19:45.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018922' 2024-09-15T02:19:45.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:19:46.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018922 -lt 128849018922 2024-09-15T02:19:46.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:46.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855392 2024-09-15T02:19:46.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:46.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:19:46.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855392 2024-09-15T02:19:46.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:46.093 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 150323855392 2024-09-15T02:19:46.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855392 2024-09-15T02:19:46.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-09-15T02:19:46.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:19:46.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855393 -lt 150323855392 2024-09-15T02:19:46.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:46.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691868 2024-09-15T02:19:46.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:46.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:19:46.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691868 2024-09-15T02:19:46.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:46.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691868 2024-09-15T02:19:46.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-09-15T02:19:46.451 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.7 seq 171798691868 2024-09-15T02:19:46.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:19:46.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691869 -lt 171798691868 2024-09-15T02:19:46.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:46.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528344 2024-09-15T02:19:46.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:46.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:19:46.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528344 2024-09-15T02:19:46.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:46.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528344 2024-09-15T02:19:46.803 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.8 seq 193273528344 2024-09-15T02:19:46.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-09-15T02:19:46.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:19:47.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528345 -lt 193273528344 2024-09-15T02:19:47.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:47.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364813 2024-09-15T02:19:47.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:47.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:19:47.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364813 2024-09-15T02:19:47.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:47.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364813 2024-09-15T02:19:47.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-09-15T02:19:47.171 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.9 seq 214748364813 2024-09-15T02:19:47.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:19:47.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364814 -lt 214748364813 2024-09-15T02:19:47.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:47.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201290 2024-09-15T02:19:47.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:47.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:19:47.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201290 2024-09-15T02:19:47.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:47.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201290 2024-09-15T02:19:47.576 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.10 seq 236223201290 2024-09-15T02:19:47.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201290' 2024-09-15T02:19:47.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:19:48.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201291 -lt 236223201290 2024-09-15T02:19:48.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:19:48.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:19:48.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:19:48.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T02:19:48.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:19:48.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:19:48.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:19:48.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:19:48.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:19:48.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:19:48.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:19:48.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T02:19:48.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:19:48.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:19:48.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:19:49.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T02:19:49.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:19:49.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.55401/ceph-osd.0.asok 2024-09-15T02:19:49.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-09-15T02:19:49.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.55401/ceph-osd.0.asok log flush 2024-09-15T02:19:49.522 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:19:49.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-09-15T02:19:49.532 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:13:30.032+0000 7ff53540a440 0 load: jerasure load: lrc 2024-09-15T02:19:49.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-09-15T02:19:49.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-09-15T02:19:49.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-09-15T02:19:50.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-09-15T02:19:50.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-09-15T02:19:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'ecpool' already exists 2024-09-15T02:19:51.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:19:52.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:19:52.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:19:52.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:19:52.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:19:52.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:19:52.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:19:52.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:19:52.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:19:52.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:19:52.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:19:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:19:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:19:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:19:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:19:52.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:19:52.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:19:53.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:19:53.085 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:19:53.085 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:53.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:19:53.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836556 2024-09-15T02:19:53.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836556 2024-09-15T02:19:53.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556' 2024-09-15T02:19:53.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:53.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:19:53.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673035 2024-09-15T02:19:53.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673035 2024-09-15T02:19:53.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035' 2024-09-15T02:19:53.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:53.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:19:53.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509507 2024-09-15T02:19:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509507 2024-09-15T02:19:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507' 2024-09-15T02:19:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:53.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:19:53.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345983 2024-09-15T02:19:53.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345983 2024-09-15T02:19:53.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983' 2024-09-15T02:19:53.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:53.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:19:54.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182455 2024-09-15T02:19:54.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182455 2024-09-15T02:19:54.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983 4-107374182455' 2024-09-15T02:19:54.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:54.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:19:54.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018926 2024-09-15T02:19:54.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018926 2024-09-15T02:19:54.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983 4-107374182455 5-128849018926' 2024-09-15T02:19:54.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:54.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:19:54.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855397 2024-09-15T02:19:54.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855397 2024-09-15T02:19:54.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983 4-107374182455 5-128849018926 6-150323855397' 2024-09-15T02:19:54.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:54.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:19:54.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691873 2024-09-15T02:19:54.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691873 2024-09-15T02:19:54.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983 4-107374182455 5-128849018926 6-150323855397 7-171798691873' 2024-09-15T02:19:54.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:54.362 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:19:54.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528348 2024-09-15T02:19:54.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528348 2024-09-15T02:19:54.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983 4-107374182455 5-128849018926 6-150323855397 7-171798691873 8-193273528348' 2024-09-15T02:19:54.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:54.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:19:54.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364817 2024-09-15T02:19:54.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364817 2024-09-15T02:19:54.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983 4-107374182455 5-128849018926 6-150323855397 7-171798691873 8-193273528348 9-214748364817' 2024-09-15T02:19:54.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:19:54.571 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:19:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201295 2024-09-15T02:19:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201295 2024-09-15T02:19:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836556 1-42949673035 2-64424509507 3-85899345983 4-107374182455 5-128849018926 6-150323855397 7-171798691873 8-193273528348 9-214748364817 10-236223201295' 2024-09-15T02:19:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:54.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836556 2024-09-15T02:19:54.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:54.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:19:54.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836556 2024-09-15T02:19:54.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:54.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836556 2024-09-15T02:19:54.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-09-15T02:19:54.680 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836556 2024-09-15T02:19:54.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:19:55.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836556 -lt 21474836556 2024-09-15T02:19:55.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:55.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673035 2024-09-15T02:19:55.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:55.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:19:55.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673035 2024-09-15T02:19:55.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:55.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673035 2024-09-15T02:19:55.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-09-15T02:19:55.052 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673035 2024-09-15T02:19:55.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:19:55.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673035 -lt 42949673035 2024-09-15T02:19:55.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:55.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509507 2024-09-15T02:19:55.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:55.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:19:55.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509507 2024-09-15T02:19:55.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:55.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509507 2024-09-15T02:19:55.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-09-15T02:19:55.406 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509507 2024-09-15T02:19:55.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:19:55.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509508 -lt 64424509507 2024-09-15T02:19:55.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:55.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345983 2024-09-15T02:19:55.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:55.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:19:55.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345983 2024-09-15T02:19:55.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:55.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345983 2024-09-15T02:19:55.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345983' 2024-09-15T02:19:55.813 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345983 2024-09-15T02:19:55.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:19:56.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345983 -lt 85899345983 2024-09-15T02:19:56.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:56.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182455 2024-09-15T02:19:56.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:56.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:19:56.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182455 2024-09-15T02:19:56.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:56.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182455 2024-09-15T02:19:56.170 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182455 2024-09-15T02:19:56.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-09-15T02:19:56.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:19:56.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182455 -lt 107374182455 2024-09-15T02:19:56.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:56.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018926 2024-09-15T02:19:56.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:56.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:19:56.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018926 2024-09-15T02:19:56.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:56.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018926 2024-09-15T02:19:56.550 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018926 2024-09-15T02:19:56.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018926' 2024-09-15T02:19:56.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:19:56.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018927 -lt 128849018926 2024-09-15T02:19:56.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:56.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855397 2024-09-15T02:19:56.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:56.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:19:56.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855397 2024-09-15T02:19:56.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855397 2024-09-15T02:19:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-09-15T02:19:56.903 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 150323855397 2024-09-15T02:19:56.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:19:57.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855397 -lt 150323855397 2024-09-15T02:19:57.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:57.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691873 2024-09-15T02:19:57.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:57.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:19:57.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691873 2024-09-15T02:19:57.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691873 2024-09-15T02:19:57.253 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.7 seq 171798691873 2024-09-15T02:19:57.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-09-15T02:19:57.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:19:57.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691873 -lt 171798691873 2024-09-15T02:19:57.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:57.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528348 2024-09-15T02:19:57.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:57.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:19:57.605 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528348 2024-09-15T02:19:57.605 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:57.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528348 2024-09-15T02:19:57.607 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.8 seq 193273528348 2024-09-15T02:19:57.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-09-15T02:19:57.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:19:57.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528349 -lt 193273528348 2024-09-15T02:19:57.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:57.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364817 2024-09-15T02:19:57.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:57.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:19:57.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364817 2024-09-15T02:19:57.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:57.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364817 2024-09-15T02:19:57.960 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.9 seq 214748364817 2024-09-15T02:19:57.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-09-15T02:19:57.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:19:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364818 -lt 214748364817 2024-09-15T02:19:58.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:19:58.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201295 2024-09-15T02:19:58.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:19:58.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:19:58.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201295 2024-09-15T02:19:58.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:19:58.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201295 2024-09-15T02:19:58.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201295' 2024-09-15T02:19:58.324 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.10 seq 236223201295 2024-09-15T02:19:58.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:19:58.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201295 -lt 236223201295 2024-09-15T02:19:58.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:19:58.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:19:58.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:19:59.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T02:19:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:19:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:19:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:19:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:19:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:19:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:19:59.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:19:59.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T02:19:59.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:19:59.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:19:59.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:20:00.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T02:20:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:20:00.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:20:00.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-09-15T02:20:00.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_chunk_mapping 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_put_get_isa 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_put_get_jerasure 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_put_get_lrc_advanced 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_put_get_lrc_kml 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_put_get_shec' 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_alignment_constraints: local payload=ABC 2024-09-15T02:20:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:250: TEST_alignment_constraints: echo ABC 2024-09-15T02:20:00.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T02:20:00.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: TEST_alignment_constraints: local stripe_unit=4096 2024-09-15T02:20:00.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-09-15T02:20:00.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: grep k= 2024-09-15T02:20:00.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: eval local k=2 2024-09-15T02:20:00.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: local k=2 2024-09-15T02:20:00.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:258: TEST_alignment_constraints: local block_size=8191 2024-09-15T02:20:00.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-09-15T02:20:00.439 INFO:tasks.workunit.client.0.smithi130.stderr:2+0 records in 2024-09-15T02:20:00.440 INFO:tasks.workunit.client.0.smithi130.stderr:2+0 records out 2024-09-15T02:20:00.440 INFO:tasks.workunit.client.0.smithi130.stderr:16382 bytes (16 kB, 16 KiB) copied, 9.3084e-05 s, 176 MB/s 2024-09-15T02:20:00.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-09-15T02:20:00.486 INFO:tasks.workunit.client.0.smithi130.stderr:INFO: op_size has been rounded to 8192 2024-09-15T02:20:00.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:278: verify_chunk_mapping: local poolname=ecpool 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: verify_chunk_mapping: local first=0 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:280: verify_chunk_mapping: local second=1 2024-09-15T02:20:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-15T02:20:00.941 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T02:20:00.953 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-15T02:20:00.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-09-15T02:20:00.954 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-15T02:20:00.954 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T02:20:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-15T02:20:00.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-09-15T02:20:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-09-15T02:20:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-09-15T02:20:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:285: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-09-15T02:20:01.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-09-15T02:20:01.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-15T02:20:01.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-15T02:20:01.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-09-15T02:20:01.300 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-15T02:20:01.300 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:20:01.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:20:01.658 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:20:01.658 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:20:01.659 INFO:tasks.workunit.client.0.smithi130.stderr:0' 2024-09-15T02:20:01.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-09-15T02:20:01.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-15T02:20:01.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: local -a osds 2024-09-15T02:20:01.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i = 0 )) 2024-09-15T02:20:01.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-15T02:20:01.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-09-15T02:20:01.740 INFO:tasks.workunit.client.0.smithi130.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T02:20:01.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-15T02:20:01.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-15T02:20:01.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-09-15T02:20:01.822 INFO:tasks.workunit.client.0.smithi130.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T02:20:01.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-15T02:20:01.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-15T02:20:01.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-15T02:20:01.905 INFO:tasks.workunit.client.0.smithi130.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T02:20:01.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-15T02:20:01.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-15T02:20:01.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-09-15T02:20:01.986 INFO:tasks.workunit.client.0.smithi130.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T02:20:01.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-15T02:20:01.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-15T02:20:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:20:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-15T02:20:01.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-15T02:20:01.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-15T02:20:01.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-09-15T02:20:01.995 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-15T02:20:01.995 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:20:02.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:0' 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: local -a osds 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-15T02:20:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:20:02.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:20:02.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:20:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-15T02:20:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T02:20:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:20:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-15T02:20:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:20:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-09-15T02:20:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-09-15T02:20:04.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-09-15T02:20:04.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:20:04.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:20:04.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:20:04.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:20:04.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:20:04.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:20:04.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:20:04.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2024-09-15T02:20:04.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T02:20:04.925 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.3 2024-09-15T02:20:04.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:20:04.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2024-09-15T02:20:04.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T02:20:04.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:20:04.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:20:04.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:20:04.948 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:04.946+0000 7f05b341c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:04.950 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:04.949+0000 7f05b341c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:04.951 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:04.950+0000 7f05b341c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:05.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T02:20:05.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:05.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:20:05.467 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:05.466+0000 7f05b341c440 -1 Falling back to public interface 2024-09-15T02:20:05.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:06.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:06.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:06.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:20:06.634 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:20:06.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:06.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:20:06.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:07.954 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:07.952+0000 7f05b341c440 -1 osd.3 74 log_to_monitors true 2024-09-15T02:20:07.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:07.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:08.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:20:08.000 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:20:08.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:08.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:20:08.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:20:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:20:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:20:09.888 INFO:tasks.workunit.client.0.smithi130.stderr:osd.3 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [20,74) [v2:127.0.0.1:6826/77098,v1:127.0.0.1:6827/77098] [v2:127.0.0.1:6828/77098,v1:127.0.0.1:6829/77098] exists,up 5c2e2d5f-3a7f-4bd6-853b-66ed7d22fdd5 2024-09-15T02:20:09.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:20:09.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:20:09.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:20:09.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:20:09.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:20:09.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:20:09.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:20:09.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:20:10.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:20:10.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:10.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:20:10.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836561 2024-09-15T02:20:10.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836561 2024-09-15T02:20:10.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561' 2024-09-15T02:20:10.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:10.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:20:10.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673040 2024-09-15T02:20:10.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673040 2024-09-15T02:20:10.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040' 2024-09-15T02:20:10.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:10.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:20:10.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509513 2024-09-15T02:20:10.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509513 2024-09-15T02:20:10.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513' 2024-09-15T02:20:10.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:10.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:20:10.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449091 2024-09-15T02:20:10.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449091 2024-09-15T02:20:10.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091' 2024-09-15T02:20:10.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:10.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:20:11.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182460 2024-09-15T02:20:11.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182460 2024-09-15T02:20:11.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091 4-107374182460' 2024-09-15T02:20:11.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:11.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:20:11.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018932 2024-09-15T02:20:11.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018932 2024-09-15T02:20:11.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091 4-107374182460 5-128849018932' 2024-09-15T02:20:11.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:11.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:20:11.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855402 2024-09-15T02:20:11.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855402 2024-09-15T02:20:11.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091 4-107374182460 5-128849018932 6-150323855402' 2024-09-15T02:20:11.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:11.265 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:20:11.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691878 2024-09-15T02:20:11.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691878 2024-09-15T02:20:11.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091 4-107374182460 5-128849018932 6-150323855402 7-171798691878' 2024-09-15T02:20:11.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:11.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:20:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528354 2024-09-15T02:20:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528354 2024-09-15T02:20:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091 4-107374182460 5-128849018932 6-150323855402 7-171798691878 8-193273528354' 2024-09-15T02:20:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:11.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:20:11.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364823 2024-09-15T02:20:11.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364823 2024-09-15T02:20:11.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091 4-107374182460 5-128849018932 6-150323855402 7-171798691878 8-193273528354 9-214748364823' 2024-09-15T02:20:11.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:11.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:20:11.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201300 2024-09-15T02:20:11.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201300 2024-09-15T02:20:11.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836561 1-42949673040 2-64424509513 3-335007449091 4-107374182460 5-128849018932 6-150323855402 7-171798691878 8-193273528354 9-214748364823 10-236223201300' 2024-09-15T02:20:11.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:11.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836561 2024-09-15T02:20:11.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:11.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:20:11.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836561 2024-09-15T02:20:11.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836561 2024-09-15T02:20:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-09-15T02:20:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 21474836561 2024-09-15T02:20:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:20:12.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836561 -lt 21474836561 2024-09-15T02:20:12.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:12.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673040 2024-09-15T02:20:12.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:20:12.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673040 2024-09-15T02:20:12.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:12.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673040 2024-09-15T02:20:12.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-15T02:20:12.109 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 42949673040 2024-09-15T02:20:12.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:20:12.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673040 -lt 42949673040 2024-09-15T02:20:12.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:12.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509513 2024-09-15T02:20:12.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:20:12.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509513 2024-09-15T02:20:12.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:12.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509513 2024-09-15T02:20:12.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-09-15T02:20:12.569 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 64424509513 2024-09-15T02:20:12.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:20:12.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509513 -lt 64424509513 2024-09-15T02:20:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:12.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-335007449091 2024-09-15T02:20:12.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:20:12.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-335007449091 2024-09-15T02:20:12.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:12.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449091 2024-09-15T02:20:12.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 335007449091' 2024-09-15T02:20:12.947 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 335007449091 2024-09-15T02:20:12.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:20:13.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449091 -lt 335007449091 2024-09-15T02:20:13.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:13.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182460 2024-09-15T02:20:13.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:13.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:20:13.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182460 2024-09-15T02:20:13.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:13.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182460 2024-09-15T02:20:13.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-09-15T02:20:13.333 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.4 seq 107374182460 2024-09-15T02:20:13.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:20:13.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182461 -lt 107374182460 2024-09-15T02:20:13.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:13.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018932 2024-09-15T02:20:13.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:13.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:20:13.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018932 2024-09-15T02:20:13.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:13.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018932 2024-09-15T02:20:13.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018932' 2024-09-15T02:20:13.691 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.5 seq 128849018932 2024-09-15T02:20:13.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:20:14.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018932 -lt 128849018932 2024-09-15T02:20:14.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:14.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855402 2024-09-15T02:20:14.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:14.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:20:14.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855402 2024-09-15T02:20:14.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:14.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855402 2024-09-15T02:20:14.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-09-15T02:20:14.067 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.6 seq 150323855402 2024-09-15T02:20:14.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:20:14.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855402 -lt 150323855402 2024-09-15T02:20:14.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:14.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691878 2024-09-15T02:20:14.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:14.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:20:14.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691878 2024-09-15T02:20:14.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:14.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691878 2024-09-15T02:20:14.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-09-15T02:20:14.422 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.7 seq 171798691878 2024-09-15T02:20:14.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:20:14.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691879 -lt 171798691878 2024-09-15T02:20:14.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:14.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528354 2024-09-15T02:20:14.782 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:14.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:20:14.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528354 2024-09-15T02:20:14.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:14.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528354 2024-09-15T02:20:14.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-09-15T02:20:14.784 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.8 seq 193273528354 2024-09-15T02:20:14.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:20:15.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528354 -lt 193273528354 2024-09-15T02:20:15.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:15.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364823 2024-09-15T02:20:15.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:15.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:20:15.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364823 2024-09-15T02:20:15.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:15.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364823 2024-09-15T02:20:15.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364823' 2024-09-15T02:20:15.147 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.9 seq 214748364823 2024-09-15T02:20:15.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:20:15.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364824 -lt 214748364823 2024-09-15T02:20:15.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:15.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201300 2024-09-15T02:20:15.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:15.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:20:15.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201300 2024-09-15T02:20:15.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:15.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201300 2024-09-15T02:20:15.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201300' 2024-09-15T02:20:15.539 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.10 seq 236223201300 2024-09-15T02:20:15.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:20:15.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201301 -lt 236223201300 2024-09-15T02:20:15.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:20:15.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:20:15.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:20:16.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:20:16.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T02:20:16.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:20:16.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:20:16.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: grep --quiet SECONDecpool 2024-09-15T02:20:17.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:20:17.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-09-15T02:20:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-09-15T02:20:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-09-15T02:20:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-15T02:20:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:20:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-15T02:20:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:20:20.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:20:20.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:20:20.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:20:20.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2024-09-15T02:20:20.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-15T02:20:20.534 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.5 2024-09-15T02:20:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:20:20.535 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2024-09-15T02:20:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-15T02:20:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:20:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:20:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:20:20.556 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:20.554+0000 7fd7e843b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:20.559 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:20.557+0000 7fd7e843b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:20.560 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:20.558+0000 7fd7e843b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:20:21.081 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:21.079+0000 7fd7e843b440 -1 Falling back to public interface 2024-09-15T02:20:21.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:20:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:20:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:20:22.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:23.598 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:23.596+0000 7fd7e843b440 -1 osd.5 80 log_to_monitors true 2024-09-15T02:20:23.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:23.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:23.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:20:23.616 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:20:23.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:23.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:20:24.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:20:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:20:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:25.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:20:25.620 INFO:tasks.workunit.client.0.smithi130.stderr:osd.5 up in weight 1 up_from 85 up_thru 85 down_at 81 last_clean_interval [30,80) [v2:127.0.0.1:6842/78795,v1:127.0.0.1:6843/78795] [v2:127.0.0.1:6844/78795,v1:127.0.0.1:6845/78795] exists,up ad685a9c-f09a-44c7-9f9c-c61d75840abb 2024-09-15T02:20:25.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:20:25.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:20:25.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:20:25.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:20:25.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:20:25.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:20:25.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:20:25.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:20:25.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:20:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:20:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:20:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:20:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:20:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:20:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:20:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:20:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:20:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:20:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:20:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:20:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:20:26.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:26.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:20:26.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836567 2024-09-15T02:20:26.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836567 2024-09-15T02:20:26.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567' 2024-09-15T02:20:26.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:26.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:20:26.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673046 2024-09-15T02:20:26.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673046 2024-09-15T02:20:26.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046' 2024-09-15T02:20:26.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:26.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:20:26.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509518 2024-09-15T02:20:26.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509518 2024-09-15T02:20:26.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518' 2024-09-15T02:20:26.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:26.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:20:26.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449096 2024-09-15T02:20:26.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449096 2024-09-15T02:20:26.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096' 2024-09-15T02:20:26.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:26.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:20:26.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182466 2024-09-15T02:20:26.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182466 2024-09-15T02:20:26.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096 4-107374182466' 2024-09-15T02:20:26.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:26.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:20:27.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=365072220163 2024-09-15T02:20:27.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 365072220163 2024-09-15T02:20:27.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096 4-107374182466 5-365072220163' 2024-09-15T02:20:27.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:27.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:20:27.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855407 2024-09-15T02:20:27.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855407 2024-09-15T02:20:27.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096 4-107374182466 5-365072220163 6-150323855407' 2024-09-15T02:20:27.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:27.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:20:27.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691883 2024-09-15T02:20:27.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691883 2024-09-15T02:20:27.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096 4-107374182466 5-365072220163 6-150323855407 7-171798691883' 2024-09-15T02:20:27.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:27.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:20:27.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528359 2024-09-15T02:20:27.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528359 2024-09-15T02:20:27.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096 4-107374182466 5-365072220163 6-150323855407 7-171798691883 8-193273528359' 2024-09-15T02:20:27.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:27.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:20:27.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364828 2024-09-15T02:20:27.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364828 2024-09-15T02:20:27.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096 4-107374182466 5-365072220163 6-150323855407 7-171798691883 8-193273528359 9-214748364828' 2024-09-15T02:20:27.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:27.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:20:27.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201305 2024-09-15T02:20:27.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201305 2024-09-15T02:20:27.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673046 2-64424509518 3-335007449096 4-107374182466 5-365072220163 6-150323855407 7-171798691883 8-193273528359 9-214748364828 10-236223201305' 2024-09-15T02:20:27.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:27.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836567 2024-09-15T02:20:27.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:27.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:20:27.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836567 2024-09-15T02:20:27.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:27.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836567 2024-09-15T02:20:27.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-15T02:20:27.533 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 21474836567 2024-09-15T02:20:27.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:20:27.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-15T02:20:27.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:27.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673046 2024-09-15T02:20:27.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:27.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:20:27.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673046 2024-09-15T02:20:27.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:27.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673046 2024-09-15T02:20:27.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-09-15T02:20:27.884 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 42949673046 2024-09-15T02:20:27.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:20:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673046 -lt 42949673046 2024-09-15T02:20:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:28.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509518 2024-09-15T02:20:28.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:28.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:20:28.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509518 2024-09-15T02:20:28.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:28.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509518 2024-09-15T02:20:28.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-15T02:20:28.250 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 64424509518 2024-09-15T02:20:28.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:20:28.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509518 -lt 64424509518 2024-09-15T02:20:28.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:28.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-335007449096 2024-09-15T02:20:28.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:28.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:20:28.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-335007449096 2024-09-15T02:20:28.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:28.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449096 2024-09-15T02:20:28.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 335007449096' 2024-09-15T02:20:28.601 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 335007449096 2024-09-15T02:20:28.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:20:28.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449096 -lt 335007449096 2024-09-15T02:20:28.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:28.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182466 2024-09-15T02:20:28.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:28.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:20:28.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182466 2024-09-15T02:20:28.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:28.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182466 2024-09-15T02:20:28.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182466' 2024-09-15T02:20:28.976 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.4 seq 107374182466 2024-09-15T02:20:28.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:20:29.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182466 -lt 107374182466 2024-09-15T02:20:29.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:29.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-365072220163 2024-09-15T02:20:29.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:29.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:20:29.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-365072220163 2024-09-15T02:20:29.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:29.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=365072220163 2024-09-15T02:20:29.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 365072220163' 2024-09-15T02:20:29.404 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.5 seq 365072220163 2024-09-15T02:20:29.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:20:29.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 365072220163 -lt 365072220163 2024-09-15T02:20:29.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:29.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855407 2024-09-15T02:20:29.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:29.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:20:29.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855407 2024-09-15T02:20:29.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:29.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855407 2024-09-15T02:20:29.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-09-15T02:20:29.762 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.6 seq 150323855407 2024-09-15T02:20:29.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:20:30.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855408 -lt 150323855407 2024-09-15T02:20:30.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:30.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691883 2024-09-15T02:20:30.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:30.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:20:30.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691883 2024-09-15T02:20:30.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:30.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691883 2024-09-15T02:20:30.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-09-15T02:20:30.168 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.7 seq 171798691883 2024-09-15T02:20:30.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:20:30.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691884 -lt 171798691883 2024-09-15T02:20:30.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:30.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528359 2024-09-15T02:20:30.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:30.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:20:30.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528359 2024-09-15T02:20:30.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:30.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528359 2024-09-15T02:20:30.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-09-15T02:20:30.517 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.8 seq 193273528359 2024-09-15T02:20:30.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:20:30.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528359 -lt 193273528359 2024-09-15T02:20:30.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:30.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364828 2024-09-15T02:20:30.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:30.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:20:30.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364828 2024-09-15T02:20:30.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:30.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364828 2024-09-15T02:20:30.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364828' 2024-09-15T02:20:30.875 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.9 seq 214748364828 2024-09-15T02:20:30.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:20:31.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364829 -lt 214748364828 2024-09-15T02:20:31.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:31.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201305 2024-09-15T02:20:31.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:31.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:20:31.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201305 2024-09-15T02:20:31.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:31.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201305 2024-09-15T02:20:31.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201305' 2024-09-15T02:20:31.237 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.10 seq 236223201305 2024-09-15T02:20:31.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:20:31.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201306 -lt 236223201305 2024-09-15T02:20:31.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:20:31.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:20:31.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:20:32.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T02:20:32.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:20:32.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:20:32.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:20:32.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:20:32.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:20:32.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:20:32.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:20:32.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T02:20:32.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:20:32.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:20:32.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:20:32.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T02:20:32.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:20:32.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:20:32.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-09-15T02:20:33.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:314: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-09-15T02:20:33.807 INFO:tasks.workunit.client.0.smithi130.stdout:crush-device-class= 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:crush-failure-domain=host 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:crush-root=default 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:k=-1 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:l=-1 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:layers=[ [ "cDD", "" ] ] 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:m=-1 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:mapping=_DD 2024-09-15T02:20:33.808 INFO:tasks.workunit.client.0.smithi130.stdout:plugin=lrc 2024-09-15T02:20:33.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:315: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-09-15T02:20:33.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-09-15T02:20:34.714 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'remap-pool' already exists 2024-09-15T02:20:34.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:20:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:323: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-09-15T02:20:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-15T02:20:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:278: verify_chunk_mapping: local poolname=remap-pool 2024-09-15T02:20:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: verify_chunk_mapping: local first=1 2024-09-15T02:20:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:280: verify_chunk_mapping: local second=2 2024-09-15T02:20:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-15T02:20:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T02:20:35.741 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-15T02:20:35.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-09-15T02:20:35.743 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-15T02:20:35.744 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-15T02:20:35.756 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-15T02:20:35.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-09-15T02:20:35.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-09-15T02:20:35.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-09-15T02:20:35.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:285: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-09-15T02:20:37.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-09-15T02:20:37.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-15T02:20:37.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-09-15T02:20:37.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-09-15T02:20:37.522 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-15T02:20:37.522 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:20:38.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-15T02:20:38.151 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:20:38.151 INFO:tasks.workunit.client.0.smithi130.stderr:9' 2024-09-15T02:20:38.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-09-15T02:20:38.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-15T02:20:38.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: local -a osds 2024-09-15T02:20:38.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i = 0 )) 2024-09-15T02:20:38.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-15T02:20:38.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-09-15T02:20:38.232 INFO:tasks.workunit.client.0.smithi130.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T02:20:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-15T02:20:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-15T02:20:38.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-09-15T02:20:38.315 INFO:tasks.workunit.client.0.smithi130.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T02:20:38.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-15T02:20:38.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-15T02:20:38.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-15T02:20:38.398 INFO:tasks.workunit.client.0.smithi130.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T02:20:38.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-15T02:20:38.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-15T02:20:38.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:20:38.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-15T02:20:38.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-15T02:20:38.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-09-15T02:20:38.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-09-15T02:20:38.401 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-15T02:20:38.402 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:20:38.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-15T02:20:38.765 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:20:38.765 INFO:tasks.workunit.client.0.smithi130.stderr:9' 2024-09-15T02:20:38.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: osds=('4' '6' '9') 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: local -a osds 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2024-09-15T02:20:38.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:20:38.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-09-15T02:20:38.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:20:38.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:20:38.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:20:38.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:20:38.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:20:40.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:20:40.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-15T02:20:40.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T02:20:40.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:20:40.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2024-09-15T02:20:40.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:20:40.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-09-15T02:20:40.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-09-15T02:20:42.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-09-15T02:20:42.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-15T02:20:42.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:20:42.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:20:42.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.6 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:20:42.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2024-09-15T02:20:42.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2024-09-15T02:20:42.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:20:42.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:20:42.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:20:42.527 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:42.525+0000 7fd04a028440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:42.529 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:42.528+0000 7fd04a028440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:42.530 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:42.529+0000 7fd04a028440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:20:42.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2024-09-15T02:20:42.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:20:43.045 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:43.044+0000 7fd04a028440 -1 Falling back to public interface 2024-09-15T02:20:43.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:44.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:44.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:44.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:20:44.265 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:20:44.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:44.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:20:44.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:45.600 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:20:45.599+0000 7fd04a028440 -1 osd.6 95 log_to_monitors true 2024-09-15T02:20:45.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:45.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:45.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:20:45.627 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:20:45.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:45.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:20:46.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:20:47.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:20:47.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:20:47.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:20:47.111 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:20:47.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:20:47.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:20:48.033 INFO:tasks.workunit.client.0.smithi130.stderr:osd.6 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [35,95) [v2:127.0.0.1:6850/80678,v1:127.0.0.1:6851/80678] [v2:127.0.0.1:6852/80678,v1:127.0.0.1:6853/80678] exists,up c6787b80-2616-456a-a3e2-25d3b0e67d39 2024-09-15T02:20:48.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:20:48.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:20:48.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:20:48.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:20:48.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:20:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:20:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:20:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:20:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:20:48.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:20:48.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:20:48.745 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:20:48.745 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:20:48.745 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:20:48.745 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:48.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:20:49.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836573 2024-09-15T02:20:49.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836573 2024-09-15T02:20:49.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573' 2024-09-15T02:20:49.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.057 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:20:49.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673052 2024-09-15T02:20:49.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673052 2024-09-15T02:20:49.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052' 2024-09-15T02:20:49.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:20:49.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509525 2024-09-15T02:20:49.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509525 2024-09-15T02:20:49.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525' 2024-09-15T02:20:49.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.302 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:20:49.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449103 2024-09-15T02:20:49.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449103 2024-09-15T02:20:49.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103' 2024-09-15T02:20:49.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:20:49.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182472 2024-09-15T02:20:49.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182472 2024-09-15T02:20:49.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103 4-107374182472' 2024-09-15T02:20:49.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:20:49.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=365072220170 2024-09-15T02:20:49.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 365072220170 2024-09-15T02:20:49.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103 4-107374182472 5-365072220170' 2024-09-15T02:20:49.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:20:49.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=433791696899 2024-09-15T02:20:49.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 433791696899 2024-09-15T02:20:49.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103 4-107374182472 5-365072220170 6-433791696899' 2024-09-15T02:20:49.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:20:49.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691890 2024-09-15T02:20:49.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691890 2024-09-15T02:20:49.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103 4-107374182472 5-365072220170 6-433791696899 7-171798691890' 2024-09-15T02:20:49.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:20:49.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528365 2024-09-15T02:20:49.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528365 2024-09-15T02:20:49.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103 4-107374182472 5-365072220170 6-433791696899 7-171798691890 8-193273528365' 2024-09-15T02:20:49.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:49.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:20:50.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364835 2024-09-15T02:20:50.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364835 2024-09-15T02:20:50.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103 4-107374182472 5-365072220170 6-433791696899 7-171798691890 8-193273528365 9-214748364835' 2024-09-15T02:20:50.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:20:50.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:20:50.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201312 2024-09-15T02:20:50.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201312 2024-09-15T02:20:50.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673052 2-64424509525 3-335007449103 4-107374182472 5-365072220170 6-433791696899 7-171798691890 8-193273528365 9-214748364835 10-236223201312' 2024-09-15T02:20:50.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:50.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836573 2024-09-15T02:20:50.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:50.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:20:50.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836573 2024-09-15T02:20:50.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:50.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836573 2024-09-15T02:20:50.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-15T02:20:50.162 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 21474836573 2024-09-15T02:20:50.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:20:50.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836571 -lt 21474836573 2024-09-15T02:20:50.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:20:51.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:20:51.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:20:51.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836573 -lt 21474836573 2024-09-15T02:20:51.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:51.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673052 2024-09-15T02:20:51.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:51.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:20:51.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673052 2024-09-15T02:20:51.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:51.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673052 2024-09-15T02:20:51.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-09-15T02:20:51.908 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 42949673052 2024-09-15T02:20:51.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:20:52.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673052 -lt 42949673052 2024-09-15T02:20:52.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:52.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509525 2024-09-15T02:20:52.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:52.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:20:52.281 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509525 2024-09-15T02:20:52.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509525 2024-09-15T02:20:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-09-15T02:20:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 64424509525 2024-09-15T02:20:52.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:20:52.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509525 -lt 64424509525 2024-09-15T02:20:52.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:52.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-335007449103 2024-09-15T02:20:52.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:52.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:20:52.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-335007449103 2024-09-15T02:20:52.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:52.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449103 2024-09-15T02:20:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 335007449103' 2024-09-15T02:20:52.638 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 335007449103 2024-09-15T02:20:52.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:20:53.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449103 -lt 335007449103 2024-09-15T02:20:53.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:53.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182472 2024-09-15T02:20:53.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:53.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:20:53.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182472 2024-09-15T02:20:53.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:53.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182472 2024-09-15T02:20:53.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-09-15T02:20:53.031 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.4 seq 107374182472 2024-09-15T02:20:53.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:20:53.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182473 -lt 107374182472 2024-09-15T02:20:53.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:53.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-365072220170 2024-09-15T02:20:53.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:53.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:20:53.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-365072220170 2024-09-15T02:20:53.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=365072220170 2024-09-15T02:20:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 365072220170' 2024-09-15T02:20:53.384 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.5 seq 365072220170 2024-09-15T02:20:53.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:20:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 365072220170 -lt 365072220170 2024-09-15T02:20:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:53.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-433791696899 2024-09-15T02:20:53.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:53.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:20:53.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-433791696899 2024-09-15T02:20:53.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:53.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=433791696899 2024-09-15T02:20:53.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 433791696899' 2024-09-15T02:20:53.737 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.6 seq 433791696899 2024-09-15T02:20:53.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:20:54.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696900 -lt 433791696899 2024-09-15T02:20:54.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:54.126 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691890 2024-09-15T02:20:54.126 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:54.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:20:54.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691890 2024-09-15T02:20:54.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:54.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691890 2024-09-15T02:20:54.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-15T02:20:54.128 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.7 seq 171798691890 2024-09-15T02:20:54.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:20:54.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691891 -lt 171798691890 2024-09-15T02:20:54.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:54.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528365 2024-09-15T02:20:54.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:54.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:20:54.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528365 2024-09-15T02:20:54.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:54.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528365 2024-09-15T02:20:54.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-15T02:20:54.482 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.8 seq 193273528365 2024-09-15T02:20:54.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:20:54.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528366 -lt 193273528365 2024-09-15T02:20:54.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:54.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364835 2024-09-15T02:20:54.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:54.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:20:54.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364835 2024-09-15T02:20:54.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:54.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364835 2024-09-15T02:20:54.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364835' 2024-09-15T02:20:54.880 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.9 seq 214748364835 2024-09-15T02:20:54.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:20:55.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364836 -lt 214748364835 2024-09-15T02:20:55.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:20:55.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201312 2024-09-15T02:20:55.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:20:55.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:20:55.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201312 2024-09-15T02:20:55.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:20:55.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201312 2024-09-15T02:20:55.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201312' 2024-09-15T02:20:55.263 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.10 seq 236223201312 2024-09-15T02:20:55.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:20:55.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201313 -lt 236223201312 2024-09-15T02:20:55.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:20:55.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:20:55.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:20:56.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-15T02:20:56.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:20:56.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:20:56.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:20:56.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:20:56.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:20:56.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:20:56.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:20:56.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-15T02:20:56.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:20:56.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:20:56.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:20:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-15T02:20:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:20:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:20:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-15T02:20:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-15T02:20:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:20:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:20:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:20:56.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:20:56.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-09-15T02:20:57.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-09-15T02:21:00.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-09-15T02:21:00.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-15T02:21:00.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:21:00.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-15T02:21:00.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:21:00.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2024-09-15T02:21:00.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:21:00.201 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:21:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2024-09-15T02:21:00.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-15T02:21:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.9 2024-09-15T02:21:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=3e16f0fe-50cd-4b4a-a4f8-8ff1e133ecd1 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.55401/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:21:00.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2024-09-15T02:21:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-15T02:21:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:21:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:21:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:21:00.226 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:21:00.225+0000 7ff863b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:21:00.229 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:21:00.228+0000 7ff863b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:21:00.230 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:21:00.229+0000 7ff863b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:21:00.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-15T02:21:00.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:21:00.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-15T02:21:00.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:21:00.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:21:00.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:21:00.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:21:00.564 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:21:00.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:21:00.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:21:00.747 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:21:00.746+0000 7ff863b35440 -1 Falling back to public interface 2024-09-15T02:21:00.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:21:01.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:21:01.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:21:01.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:21:01.915 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:01.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:21:01.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:21:02.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:21:03.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:21:03.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:21:03.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:21:03.273 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:21:03.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:21:03.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:21:03.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:21:03.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:21:03.642+0000 7ff863b35440 -1 osd.9 104 log_to_monitors true 2024-09-15T02:21:04.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:21:04.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:21:04.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:21:04.637 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:21:04.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:21:04.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-15T02:21:05.134 INFO:tasks.workunit.client.0.smithi130.stderr:osd.9 up in weight 1 up_from 109 up_thru 109 down_at 105 last_clean_interval [50,104) [v2:127.0.0.1:6874/82399,v1:127.0.0.1:6875/82399] [v2:127.0.0.1:6876/82399,v1:127.0.0.1:6877/82399] exists,up 30b27c15-d1a2-48cc-9f0c-967dcae995fe 2024-09-15T02:21:05.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:21:05.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:21:05.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:21:05.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:21:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:21:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:21:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:21:05.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:21:05.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:21:05.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:21:05.887 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:21:05.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:21:05.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:05.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:21:06.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836579 2024-09-15T02:21:06.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836579 2024-09-15T02:21:06.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579' 2024-09-15T02:21:06.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:06.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:21:06.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673058 2024-09-15T02:21:06.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673058 2024-09-15T02:21:06.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058' 2024-09-15T02:21:06.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:06.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:21:06.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509530 2024-09-15T02:21:06.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509530 2024-09-15T02:21:06.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530' 2024-09-15T02:21:06.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:06.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:21:06.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449108 2024-09-15T02:21:06.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449108 2024-09-15T02:21:06.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108' 2024-09-15T02:21:06.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:06.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:21:07.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182478 2024-09-15T02:21:07.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182478 2024-09-15T02:21:07.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108 4-107374182478' 2024-09-15T02:21:07.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:07.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:21:07.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=365072220175 2024-09-15T02:21:07.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 365072220175 2024-09-15T02:21:07.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108 4-107374182478 5-365072220175' 2024-09-15T02:21:07.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:07.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:21:07.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=433791696905 2024-09-15T02:21:07.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 433791696905 2024-09-15T02:21:07.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108 4-107374182478 5-365072220175 6-433791696905' 2024-09-15T02:21:07.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:07.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:21:07.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691896 2024-09-15T02:21:07.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691896 2024-09-15T02:21:07.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108 4-107374182478 5-365072220175 6-433791696905 7-171798691896' 2024-09-15T02:21:07.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:07.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:21:07.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528371 2024-09-15T02:21:07.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528371 2024-09-15T02:21:07.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108 4-107374182478 5-365072220175 6-433791696905 7-171798691896 8-193273528371' 2024-09-15T02:21:07.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:07.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:21:07.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=468151435267 2024-09-15T02:21:07.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 468151435267 2024-09-15T02:21:07.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108 4-107374182478 5-365072220175 6-433791696905 7-171798691896 8-193273528371 9-468151435267' 2024-09-15T02:21:07.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:07.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:21:07.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201317 2024-09-15T02:21:07.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201317 2024-09-15T02:21:07.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836579 1-42949673058 2-64424509530 3-335007449108 4-107374182478 5-365072220175 6-433791696905 7-171798691896 8-193273528371 9-468151435267 10-236223201317' 2024-09-15T02:21:07.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:07.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836579 2024-09-15T02:21:07.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:07.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:21:07.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836579 2024-09-15T02:21:07.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:07.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836579 2024-09-15T02:21:07.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-15T02:21:07.900 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 21474836579 2024-09-15T02:21:07.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:21:08.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836579 -lt 21474836579 2024-09-15T02:21:08.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:08.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673058 2024-09-15T02:21:08.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:08.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:21:08.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673058 2024-09-15T02:21:08.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:08.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673058 2024-09-15T02:21:08.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-09-15T02:21:08.251 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 42949673058 2024-09-15T02:21:08.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:21:08.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673058 -lt 42949673058 2024-09-15T02:21:08.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:08.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509530 2024-09-15T02:21:08.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:08.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:21:08.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509530 2024-09-15T02:21:08.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:08.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509530 2024-09-15T02:21:08.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-15T02:21:08.777 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 64424509530 2024-09-15T02:21:08.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:21:09.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509530 -lt 64424509530 2024-09-15T02:21:09.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:09.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-335007449108 2024-09-15T02:21:09.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:09.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:21:09.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-335007449108 2024-09-15T02:21:09.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449108 2024-09-15T02:21:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 335007449108' 2024-09-15T02:21:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 335007449108 2024-09-15T02:21:09.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:21:09.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449109 -lt 335007449108 2024-09-15T02:21:09.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:09.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182478 2024-09-15T02:21:09.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:09.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:21:09.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182478 2024-09-15T02:21:09.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:09.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182478 2024-09-15T02:21:09.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-09-15T02:21:09.523 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.4 seq 107374182478 2024-09-15T02:21:09.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:21:09.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182478 -lt 107374182478 2024-09-15T02:21:09.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:09.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-365072220175 2024-09-15T02:21:09.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:09.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:21:09.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-365072220175 2024-09-15T02:21:09.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:09.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=365072220175 2024-09-15T02:21:09.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 365072220175' 2024-09-15T02:21:09.916 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.5 seq 365072220175 2024-09-15T02:21:09.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:21:10.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 365072220175 -lt 365072220175 2024-09-15T02:21:10.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:10.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-433791696905 2024-09-15T02:21:10.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:10.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:21:10.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-433791696905 2024-09-15T02:21:10.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:10.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=433791696905 2024-09-15T02:21:10.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 433791696905' 2024-09-15T02:21:10.265 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.6 seq 433791696905 2024-09-15T02:21:10.265 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:21:10.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696905 -lt 433791696905 2024-09-15T02:21:10.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:10.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691896 2024-09-15T02:21:10.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:10.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:21:10.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691896 2024-09-15T02:21:10.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691896 2024-09-15T02:21:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-09-15T02:21:10.616 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.7 seq 171798691896 2024-09-15T02:21:10.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:21:11.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691896 -lt 171798691896 2024-09-15T02:21:11.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:11.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528371 2024-09-15T02:21:11.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:11.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:21:11.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528371 2024-09-15T02:21:11.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:11.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528371 2024-09-15T02:21:11.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-09-15T02:21:11.004 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.8 seq 193273528371 2024-09-15T02:21:11.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:21:11.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528371 -lt 193273528371 2024-09-15T02:21:11.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:11.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-468151435267 2024-09-15T02:21:11.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:11.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:21:11.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-468151435267 2024-09-15T02:21:11.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:11.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=468151435267 2024-09-15T02:21:11.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 468151435267' 2024-09-15T02:21:11.406 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.9 seq 468151435267 2024-09-15T02:21:11.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:21:11.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 468151435268 -lt 468151435267 2024-09-15T02:21:11.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:11.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201317 2024-09-15T02:21:11.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:11.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:21:11.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201317 2024-09-15T02:21:11.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:11.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201317 2024-09-15T02:21:11.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201317' 2024-09-15T02:21:11.758 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.10 seq 236223201317 2024-09-15T02:21:11.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:21:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201318 -lt 236223201317 2024-09-15T02:21:12.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:21:12.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:21:12.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:21:12.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:21:12.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-15T02:21:12.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:21:12.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:21:12.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:21:13.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-15T02:21:13.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:21:13.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:21:13.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:325: TEST_chunk_mapping: delete_pool remap-pool 2024-09-15T02:21:13.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2024-09-15T02:21:13.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-09-15T02:21:13.865 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'remap-pool' does not exist 2024-09-15T02:21:13.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:326: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-09-15T02:21:15.048 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile remap-profile does not exist 2024-09-15T02:21:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T02:21:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-09-15T02:21:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-09-15T02:21:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: erasure_code_plugin_exists: local plugin=isa 2024-09-15T02:21:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: erasure_code_plugin_exists: local status 2024-09-15T02:21:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: local grepstr 2024-09-15T02:21:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: erasure_code_plugin_exists: local s 2024-09-15T02:21:15.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: case `uname` in 2024-09-15T02:21:15.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: uname 2024-09-15T02:21:15.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-09-15T02:21:15.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-09-15T02:21:15.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: s= 2024-09-15T02:21:15.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: erasure_code_plugin_exists: local status=0 2024-09-15T02:21:15.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-15T02:21:15.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-15T02:21:16.192 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-15T02:21:16.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: erasure_code_plugin_exists: return 0 2024-09-15T02:21:16.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-09-15T02:21:16.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:194: TEST_rados_put_get_isa: local poolname=pool-isa 2024-09-15T02:21:16.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:196: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-09-15T02:21:16.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:199: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-09-15T02:21:16.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-09-15T02:21:17.807 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-isa' already exists 2024-09-15T02:21:17.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:21:18.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:202: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-09-15T02:21:18.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T02:21:18.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-09-15T02:21:18.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T02:21:18.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T02:21:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T02:21:20.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:21:20.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:21:20.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T02:21:20.325 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-09-15T02:21:20.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-09-15T02:21:20.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:21:20.326 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-15T02:21:20.326 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:10 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:9' 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-09-15T02:21:20.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-09-15T02:21:20.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T02:21:20.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-09-15T02:21:20.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-15T02:21:21.375 INFO:tasks.workunit.client.0.smithi130.stderr:osd.9 is already out. 2024-09-15T02:21:21.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-isa SOMETHING 2024-09-15T02:21:21.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-09-15T02:21:21.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:21:21.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<9\>' 2024-09-15T02:21:21.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-15T02:21:21.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:21:21.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:10 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-09-15T02:21:21.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:21:22.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:21:22.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 9 2024-09-15T02:21:24.092 INFO:tasks.workunit.client.0.smithi130.stderr:osd.9 is already in. 2024-09-15T02:21:24.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:21:24.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:204: TEST_rados_put_get_isa: delete_pool pool-isa 2024-09-15T02:21:24.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2024-09-15T02:21:24.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-09-15T02:21:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-isa' does not exist 2024-09-15T02:21:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T02:21:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-09-15T02:21:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-09-15T02:21:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:210: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T02:21:25.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T02:21:29.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:21:29.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:21:29.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T02:21:29.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-09-15T02:21:29.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-15T02:21:29.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:21:29.512 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T02:21:29.512 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:21:30.077 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-09-15T02:21:30.077 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:21:30.077 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:21:30.077 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-15T02:21:30.077 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2024-09-15T02:21:30.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '5') 2024-09-15T02:21:30.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T02:21:30.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-09-15T02:21:30.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 5 2024-09-15T02:21:31.166 INFO:tasks.workunit.client.0.smithi130.stderr:osd.5 is already out. 2024-09-15T02:21:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds ecpool SOMETHING 2024-09-15T02:21:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-15T02:21:31.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:21:31.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<5\>' 2024-09-15T02:21:31.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-15T02:21:31.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:21:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-09-15T02:21:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:21:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:21:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:9' 2024-09-15T02:21:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2024-09-15T02:21:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:21:32.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:21:32.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 5 2024-09-15T02:21:33.446 INFO:tasks.workunit.client.0.smithi130.stderr:osd.5 is already in. 2024-09-15T02:21:33.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:21:33.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-09-15T02:21:33.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:213: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-09-15T02:21:33.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:215: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-09-15T02:21:34.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-09-15T02:21:34.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-09-15T02:21:36.857 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:21:36.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:21:37.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:222: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T02:21:37.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T02:21:39.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:21:39.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:21:39.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T02:21:39.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-15T02:21:39.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:21:39.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:21:39.483 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T02:21:39.483 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T02:21:40.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-09-15T02:21:40.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-09-15T02:21:41.531 INFO:tasks.workunit.client.0.smithi130.stderr:osd.10 is already out. 2024-09-15T02:21:41.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-15T02:21:41.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:21:41.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:21:41.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<10\>' 2024-09-15T02:21:41.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T02:21:41.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:21:42.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-15T02:21:42.244 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:42.244 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:21:42.244 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:21:42.244 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:21:42.244 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:21:42.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2024-09-15T02:21:42.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:21:42.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:21:42.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 10 2024-09-15T02:21:44.875 INFO:tasks.workunit.client.0.smithi130.stderr:osd.10 is already in. 2024-09-15T02:21:44.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:21:44.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_osds_out_in: local dir=td/test-erasure-code 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:101: rados_osds_out_in: local poolname=pool-jerasure 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:102: rados_osds_out_in: local objname=SOMETHING 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 IIII 2024-09-15T02:21:44.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:112: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T02:21:47.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:113: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:21:48.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:114: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:21:48.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:115: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-09-15T02:21:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:123: rados_osds_out_in: wait_for_clean 2024-09-15T02:21:48.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:21:48.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:21:48.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:21:48.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:21:48.035 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:21:48.036 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:21:48.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:21:48.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:21:48.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:21:48.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:21:48.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:21:48.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:21:48.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:21:48.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:21:48.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:21:48.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:21:48.615 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:48.615 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:21:48.615 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:21:48.615 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:48.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:21:48.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836589 2024-09-15T02:21:48.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836589 2024-09-15T02:21:48.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589' 2024-09-15T02:21:48.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:48.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:21:49.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673068 2024-09-15T02:21:49.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673068 2024-09-15T02:21:49.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068' 2024-09-15T02:21:49.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:49.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:21:49.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509541 2024-09-15T02:21:49.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509541 2024-09-15T02:21:49.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541' 2024-09-15T02:21:49.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:49.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:21:49.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449119 2024-09-15T02:21:49.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449119 2024-09-15T02:21:49.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119' 2024-09-15T02:21:49.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:49.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:21:49.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182488 2024-09-15T02:21:49.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182488 2024-09-15T02:21:49.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119 4-107374182488' 2024-09-15T02:21:49.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:49.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:21:49.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=365072220186 2024-09-15T02:21:49.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 365072220186 2024-09-15T02:21:49.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119 4-107374182488 5-365072220186' 2024-09-15T02:21:49.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:49.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:21:50.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=433791696915 2024-09-15T02:21:50.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 433791696915 2024-09-15T02:21:50.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119 4-107374182488 5-365072220186 6-433791696915' 2024-09-15T02:21:50.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:50.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:21:50.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691906 2024-09-15T02:21:50.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691906 2024-09-15T02:21:50.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119 4-107374182488 5-365072220186 6-433791696915 7-171798691906' 2024-09-15T02:21:50.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:50.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:21:50.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528381 2024-09-15T02:21:50.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528381 2024-09-15T02:21:50.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119 4-107374182488 5-365072220186 6-433791696915 7-171798691906 8-193273528381' 2024-09-15T02:21:50.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:50.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:21:50.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=468151435278 2024-09-15T02:21:50.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 468151435278 2024-09-15T02:21:50.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119 4-107374182488 5-365072220186 6-433791696915 7-171798691906 8-193273528381 9-468151435278' 2024-09-15T02:21:50.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:21:50.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:21:50.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201328 2024-09-15T02:21:50.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201328 2024-09-15T02:21:50.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836589 1-42949673068 2-64424509541 3-335007449119 4-107374182488 5-365072220186 6-433791696915 7-171798691906 8-193273528381 9-468151435278 10-236223201328' 2024-09-15T02:21:50.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:50.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836589 2024-09-15T02:21:50.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:50.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:21:50.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836589 2024-09-15T02:21:50.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:50.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836589 2024-09-15T02:21:50.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-09-15T02:21:50.884 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836589 2024-09-15T02:21:50.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:21:51.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836589 -lt 21474836589 2024-09-15T02:21:51.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:51.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673068 2024-09-15T02:21:51.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:51.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:21:51.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673068 2024-09-15T02:21:51.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:51.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673068 2024-09-15T02:21:51.254 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673068 2024-09-15T02:21:51.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2024-09-15T02:21:51.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:21:51.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673068 -lt 42949673068 2024-09-15T02:21:51.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:51.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509541 2024-09-15T02:21:51.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:51.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:21:51.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509541 2024-09-15T02:21:51.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:51.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509541 2024-09-15T02:21:51.857 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509541 2024-09-15T02:21:51.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2024-09-15T02:21:51.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:21:52.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509541 -lt 64424509541 2024-09-15T02:21:52.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:52.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-335007449119 2024-09-15T02:21:52.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:52.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:21:52.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-335007449119 2024-09-15T02:21:52.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:52.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449119 2024-09-15T02:21:52.213 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 335007449119 2024-09-15T02:21:52.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 335007449119' 2024-09-15T02:21:52.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:21:52.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449119 -lt 335007449119 2024-09-15T02:21:52.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:52.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182488 2024-09-15T02:21:52.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:52.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:21:52.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182488 2024-09-15T02:21:52.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:52.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182488 2024-09-15T02:21:52.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182488' 2024-09-15T02:21:52.598 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182488 2024-09-15T02:21:52.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:21:52.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182489 -lt 107374182488 2024-09-15T02:21:52.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:52.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-365072220186 2024-09-15T02:21:52.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:52.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:21:52.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-365072220186 2024-09-15T02:21:52.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:52.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=365072220186 2024-09-15T02:21:52.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 365072220186' 2024-09-15T02:21:52.981 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 365072220186 2024-09-15T02:21:52.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:21:53.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 365072220186 -lt 365072220186 2024-09-15T02:21:53.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:53.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-433791696915 2024-09-15T02:21:53.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:53.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:21:53.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-433791696915 2024-09-15T02:21:53.367 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:53.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=433791696915 2024-09-15T02:21:53.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 433791696915' 2024-09-15T02:21:53.368 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 433791696915 2024-09-15T02:21:53.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:21:53.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696916 -lt 433791696915 2024-09-15T02:21:53.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:53.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691906 2024-09-15T02:21:53.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:53.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:21:53.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691906 2024-09-15T02:21:53.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691906 2024-09-15T02:21:53.776 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.7 seq 171798691906 2024-09-15T02:21:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-09-15T02:21:53.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:21:54.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691907 -lt 171798691906 2024-09-15T02:21:54.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:54.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528381 2024-09-15T02:21:54.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:54.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:21:54.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528381 2024-09-15T02:21:54.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:54.197 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.8 seq 193273528381 2024-09-15T02:21:54.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528381 2024-09-15T02:21:54.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528381' 2024-09-15T02:21:54.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:21:54.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528382 -lt 193273528381 2024-09-15T02:21:54.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:54.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-468151435278 2024-09-15T02:21:54.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:54.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:21:54.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-468151435278 2024-09-15T02:21:54.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:54.548 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.9 seq 468151435278 2024-09-15T02:21:54.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=468151435278 2024-09-15T02:21:54.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 468151435278' 2024-09-15T02:21:54.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:21:54.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 468151435279 -lt 468151435278 2024-09-15T02:21:54.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:21:54.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201328 2024-09-15T02:21:54.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:21:54.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:21:54.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201328 2024-09-15T02:21:54.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:21:54.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201328 2024-09-15T02:21:54.900 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.10 seq 236223201328 2024-09-15T02:21:54.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201328' 2024-09-15T02:21:54.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:21:55.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201329 -lt 236223201328 2024-09-15T02:21:55.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:21:55.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:21:55.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:21:55.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-15T02:21:55.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:21:55.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:21:55.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:21:55.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:21:55.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:21:55.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:21:55.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:21:56.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=27 2024-09-15T02:21:56.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:21:56.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:21:56.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:21:56.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 27 = 28 2024-09-15T02:21:56.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 27 '!=' -1 2024-09-15T02:21:56.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T02:21:56.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=27 2024-09-15T02:21:56.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:21:56.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:21:56.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:21:57.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-15T02:21:57.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:21:57.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:21:57.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:21:57.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-15T02:21:57.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:21:57.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:21:57.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:124: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T02:21:57.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:21:57.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:21:57.577 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T02:21:57.577 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:21:57.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-15T02:21:57.971 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:124: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:125: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:125: rados_osds_out_in: local -a osds 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:126: rados_osds_out_in: for osd in 0 1 2024-09-15T02:21:57.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: ceph osd out 5 2024-09-15T02:21:58.454 INFO:tasks.workunit.client.0.smithi130.stderr:osd.5 is already out. 2024-09-15T02:21:58.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:126: rados_osds_out_in: for osd in 0 1 2024-09-15T02:21:58.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: ceph osd out 1 2024-09-15T02:22:00.202 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. 2024-09-15T02:22:00.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: wait_for_clean 2024-09-15T02:22:00.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:22:00.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:22:00.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:22:00.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:22:00.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:22:00.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:22:00.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:22:00.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:22:00.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:22:00.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:22:00.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:22:00.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:22:00.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:22:00.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:22:00.300 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:22:00.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:22:00.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:00.862 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:22:01.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836594 2024-09-15T02:22:01.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836594 2024-09-15T02:22:01.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594' 2024-09-15T02:22:01.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:01.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:22:01.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673073 2024-09-15T02:22:01.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673073 2024-09-15T02:22:01.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073' 2024-09-15T02:22:01.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:01.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:22:01.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509545 2024-09-15T02:22:01.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509545 2024-09-15T02:22:01.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545' 2024-09-15T02:22:01.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:01.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:22:01.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449123 2024-09-15T02:22:01.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449123 2024-09-15T02:22:01.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123' 2024-09-15T02:22:01.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:01.731 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:22:02.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182493 2024-09-15T02:22:02.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182493 2024-09-15T02:22:02.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123 4-107374182493' 2024-09-15T02:22:02.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:02.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:22:02.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=365072220190 2024-09-15T02:22:02.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 365072220190 2024-09-15T02:22:02.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123 4-107374182493 5-365072220190' 2024-09-15T02:22:02.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:02.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:22:02.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=433791696920 2024-09-15T02:22:02.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 433791696920 2024-09-15T02:22:02.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123 4-107374182493 5-365072220190 6-433791696920' 2024-09-15T02:22:02.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:02.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:22:02.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691910 2024-09-15T02:22:02.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691910 2024-09-15T02:22:02.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123 4-107374182493 5-365072220190 6-433791696920 7-171798691910' 2024-09-15T02:22:02.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:02.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:22:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528386 2024-09-15T02:22:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528386 2024-09-15T02:22:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123 4-107374182493 5-365072220190 6-433791696920 7-171798691910 8-193273528386' 2024-09-15T02:22:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:02.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:22:02.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=468151435282 2024-09-15T02:22:02.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 468151435282 2024-09-15T02:22:02.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123 4-107374182493 5-365072220190 6-433791696920 7-171798691910 8-193273528386 9-468151435282' 2024-09-15T02:22:02.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:02.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:22:03.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201332 2024-09-15T02:22:03.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201332 2024-09-15T02:22:03.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836594 1-42949673073 2-64424509545 3-335007449123 4-107374182493 5-365072220190 6-433791696920 7-171798691910 8-193273528386 9-468151435282 10-236223201332' 2024-09-15T02:22:03.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:03.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836594 2024-09-15T02:22:03.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:03.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:22:03.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836594 2024-09-15T02:22:03.085 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:03.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836594 2024-09-15T02:22:03.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2024-09-15T02:22:03.086 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836594 2024-09-15T02:22:03.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:22:03.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836594 -lt 21474836594 2024-09-15T02:22:03.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:03.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673073 2024-09-15T02:22:03.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:03.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:22:03.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673073 2024-09-15T02:22:03.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:03.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673073 2024-09-15T02:22:03.807 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673073 2024-09-15T02:22:03.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673073' 2024-09-15T02:22:03.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:22:04.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673073 -lt 42949673073 2024-09-15T02:22:04.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:04.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509545 2024-09-15T02:22:04.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:04.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:22:04.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509545 2024-09-15T02:22:04.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:04.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509545 2024-09-15T02:22:04.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509545' 2024-09-15T02:22:04.348 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509545 2024-09-15T02:22:04.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:22:04.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509545 -lt 64424509545 2024-09-15T02:22:04.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:04.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-335007449123 2024-09-15T02:22:04.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:04.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:22:04.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-335007449123 2024-09-15T02:22:04.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:04.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449123 2024-09-15T02:22:04.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 335007449123' 2024-09-15T02:22:04.741 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 335007449123 2024-09-15T02:22:04.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:22:05.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449124 -lt 335007449123 2024-09-15T02:22:05.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:05.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182493 2024-09-15T02:22:05.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:05.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:22:05.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182493 2024-09-15T02:22:05.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:05.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182493 2024-09-15T02:22:05.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-09-15T02:22:05.095 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182493 2024-09-15T02:22:05.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:22:05.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182493 -lt 107374182493 2024-09-15T02:22:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:05.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-365072220190 2024-09-15T02:22:05.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:05.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:22:05.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-365072220190 2024-09-15T02:22:05.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:05.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=365072220190 2024-09-15T02:22:05.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 365072220190' 2024-09-15T02:22:05.473 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 365072220190 2024-09-15T02:22:05.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:22:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 365072220191 -lt 365072220190 2024-09-15T02:22:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:05.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-433791696920 2024-09-15T02:22:05.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:05.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:22:05.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-433791696920 2024-09-15T02:22:05.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:05.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=433791696920 2024-09-15T02:22:05.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 433791696920' 2024-09-15T02:22:05.826 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 433791696920 2024-09-15T02:22:05.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:22:06.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696920 -lt 433791696920 2024-09-15T02:22:06.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:06.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691910 2024-09-15T02:22:06.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:06.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:22:06.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691910 2024-09-15T02:22:06.243 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:06.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691910 2024-09-15T02:22:06.244 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.7 seq 171798691910 2024-09-15T02:22:06.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-09-15T02:22:06.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:22:06.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691911 -lt 171798691910 2024-09-15T02:22:06.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:06.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528386 2024-09-15T02:22:06.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:06.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:22:06.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528386 2024-09-15T02:22:06.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:06.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528386 2024-09-15T02:22:06.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528386' 2024-09-15T02:22:06.675 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.8 seq 193273528386 2024-09-15T02:22:06.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:22:07.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528387 -lt 193273528386 2024-09-15T02:22:07.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:07.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-468151435282 2024-09-15T02:22:07.047 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:07.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:22:07.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-468151435282 2024-09-15T02:22:07.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:07.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=468151435282 2024-09-15T02:22:07.049 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.9 seq 468151435282 2024-09-15T02:22:07.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 468151435282' 2024-09-15T02:22:07.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:22:07.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 468151435283 -lt 468151435282 2024-09-15T02:22:07.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:07.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201332 2024-09-15T02:22:07.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:07.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:22:07.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201332 2024-09-15T02:22:07.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:07.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201332 2024-09-15T02:22:07.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201332' 2024-09-15T02:22:07.420 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.10 seq 236223201332 2024-09-15T02:22:07.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:22:07.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201333 -lt 236223201332 2024-09-15T02:22:07.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:22:07.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:22:07.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:22:08.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:22:08.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=27 2024-09-15T02:22:08.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:22:08.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:22:08.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:22:09.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 27 = 28 2024-09-15T02:22:09.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 27 '!=' -1 2024-09-15T02:22:09.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T02:22:09.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=27 2024-09-15T02:22:09.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:22:09.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:22:09.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:22:09.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:22:09.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:22:09.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:22:09.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:22:09.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:22:09.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:22:09.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:22:09.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=27 2024-09-15T02:22:09.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:22:09.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:22:09.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 27 = 28 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 27 '!=' 27 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T02:22:09.994 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T02:22:10.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1023 2024-09-15T02:22:10.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1023 '!=' null 2024-09-15T02:22:10.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T02:22:10.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:22:10.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:22:10.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:22:10.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:22:10.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:22:10.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:22:10.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:22:10.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:22:10.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:22:10.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:22:10.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-15T02:22:10.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:22:10.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:22:10.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:22:11.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: for osd in 0 1 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: grep '\<5\>' 2024-09-15T02:22:11.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T02:22:11.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:11.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: for osd in 0 1 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:11.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: grep '\<1\>' 2024-09-15T02:22:11.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T02:22:11.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T02:22:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:22:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:22:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:22:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:22:12.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-09-15T02:22:12.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:136: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:22:12.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:22:12.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:143: rados_osds_out_in: for osd in 0 1 2024-09-15T02:22:12.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:144: rados_osds_out_in: ceph osd in 5 2024-09-15T02:22:12.675 INFO:tasks.workunit.client.0.smithi130.stderr:osd.5 is already in. 2024-09-15T02:22:12.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:143: rados_osds_out_in: for osd in 0 1 2024-09-15T02:22:12.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:144: rados_osds_out_in: ceph osd in 1 2024-09-15T02:22:14.451 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already in. 2024-09-15T02:22:14.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:146: rados_osds_out_in: wait_for_clean 2024-09-15T02:22:14.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:22:14.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:22:14.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:22:14.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:22:14.464 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:22:14.464 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:22:14.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:22:14.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:22:14.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:22:14.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:22:14.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:22:14.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:22:14.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:22:14.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:22:14.550 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:22:15.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:22:15.424 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:22:15.424 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:22:15.424 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:8 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:15.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:22:15.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836598 2024-09-15T02:22:15.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836598 2024-09-15T02:22:15.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598' 2024-09-15T02:22:15.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:15.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:22:16.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673078 2024-09-15T02:22:16.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673078 2024-09-15T02:22:16.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078' 2024-09-15T02:22:16.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:16.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:22:16.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509550 2024-09-15T02:22:16.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509550 2024-09-15T02:22:16.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550' 2024-09-15T02:22:16.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:16.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:22:16.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=335007449128 2024-09-15T02:22:16.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 335007449128 2024-09-15T02:22:16.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128' 2024-09-15T02:22:16.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:16.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:22:16.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182498 2024-09-15T02:22:16.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182498 2024-09-15T02:22:16.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128 4-107374182498' 2024-09-15T02:22:16.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:16.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:22:16.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=365072220195 2024-09-15T02:22:16.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 365072220195 2024-09-15T02:22:16.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128 4-107374182498 5-365072220195' 2024-09-15T02:22:16.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:16.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:22:17.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=433791696925 2024-09-15T02:22:17.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 433791696925 2024-09-15T02:22:17.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128 4-107374182498 5-365072220195 6-433791696925' 2024-09-15T02:22:17.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:17.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:22:17.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691916 2024-09-15T02:22:17.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691916 2024-09-15T02:22:17.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128 4-107374182498 5-365072220195 6-433791696925 7-171798691916' 2024-09-15T02:22:17.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:17.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:22:17.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528391 2024-09-15T02:22:17.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528391 2024-09-15T02:22:17.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128 4-107374182498 5-365072220195 6-433791696925 7-171798691916 8-193273528391' 2024-09-15T02:22:17.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:17.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-15T02:22:18.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=468151435287 2024-09-15T02:22:18.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 468151435287 2024-09-15T02:22:18.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128 4-107374182498 5-365072220195 6-433791696925 7-171798691916 8-193273528391 9-468151435287' 2024-09-15T02:22:18.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:22:18.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-15T02:22:18.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201337 2024-09-15T02:22:18.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201337 2024-09-15T02:22:18.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673078 2-64424509550 3-335007449128 4-107374182498 5-365072220195 6-433791696925 7-171798691916 8-193273528391 9-468151435287 10-236223201337' 2024-09-15T02:22:18.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:18.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836598 2024-09-15T02:22:18.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:18.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:22:18.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836598 2024-09-15T02:22:18.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:18.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836598 2024-09-15T02:22:18.131 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836598 2024-09-15T02:22:18.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-09-15T02:22:18.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:22:18.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836599 -lt 21474836598 2024-09-15T02:22:18.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:18.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673078 2024-09-15T02:22:18.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:18.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:22:18.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673078 2024-09-15T02:22:18.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673078 2024-09-15T02:22:18.493 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673078 2024-09-15T02:22:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673078' 2024-09-15T02:22:18.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:22:18.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673078 -lt 42949673078 2024-09-15T02:22:18.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:18.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509550 2024-09-15T02:22:18.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:18.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:22:18.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509550 2024-09-15T02:22:18.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:18.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509550 2024-09-15T02:22:18.909 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509550 2024-09-15T02:22:18.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509550' 2024-09-15T02:22:18.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:22:19.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509550 -lt 64424509550 2024-09-15T02:22:19.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:19.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-335007449128 2024-09-15T02:22:19.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:19.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:22:19.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-335007449128 2024-09-15T02:22:19.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:19.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=335007449128 2024-09-15T02:22:19.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 335007449128' 2024-09-15T02:22:19.264 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 335007449128 2024-09-15T02:22:19.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:22:19.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 335007449129 -lt 335007449128 2024-09-15T02:22:19.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:19.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182498 2024-09-15T02:22:19.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:19.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:22:19.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182498 2024-09-15T02:22:19.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:19.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182498 2024-09-15T02:22:19.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182498' 2024-09-15T02:22:19.691 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182498 2024-09-15T02:22:19.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:22:20.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182498 -lt 107374182498 2024-09-15T02:22:20.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:20.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-365072220195 2024-09-15T02:22:20.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:20.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:22:20.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-365072220195 2024-09-15T02:22:20.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:20.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=365072220195 2024-09-15T02:22:20.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 365072220195' 2024-09-15T02:22:20.083 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 365072220195 2024-09-15T02:22:20.083 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:22:20.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 365072220195 -lt 365072220195 2024-09-15T02:22:20.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:20.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-433791696925 2024-09-15T02:22:20.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:20.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:22:20.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-433791696925 2024-09-15T02:22:20.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:20.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=433791696925 2024-09-15T02:22:20.436 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 433791696925 2024-09-15T02:22:20.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 433791696925' 2024-09-15T02:22:20.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:22:20.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 433791696925 -lt 433791696925 2024-09-15T02:22:20.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:20.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691916 2024-09-15T02:22:20.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:20.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:22:20.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691916 2024-09-15T02:22:20.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:20.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691916 2024-09-15T02:22:20.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691916' 2024-09-15T02:22:20.790 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.7 seq 171798691916 2024-09-15T02:22:20.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:22:21.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691916 -lt 171798691916 2024-09-15T02:22:21.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:21.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528391 2024-09-15T02:22:21.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:21.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:22:21.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528391 2024-09-15T02:22:21.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:21.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528391 2024-09-15T02:22:21.146 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.8 seq 193273528391 2024-09-15T02:22:21.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528391' 2024-09-15T02:22:21.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:22:21.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528391 -lt 193273528391 2024-09-15T02:22:21.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:21.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-468151435287 2024-09-15T02:22:21.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:21.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-15T02:22:21.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-468151435287 2024-09-15T02:22:21.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:21.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=468151435287 2024-09-15T02:22:21.498 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.9 seq 468151435287 2024-09-15T02:22:21.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 468151435287' 2024-09-15T02:22:21.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-15T02:22:21.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 468151435288 -lt 468151435287 2024-09-15T02:22:21.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:22:21.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201337 2024-09-15T02:22:21.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:22:21.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-15T02:22:21.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201337 2024-09-15T02:22:21.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:22:21.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201337 2024-09-15T02:22:21.899 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.10 seq 236223201337 2024-09-15T02:22:21.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201337' 2024-09-15T02:22:21.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-15T02:22:22.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201338 -lt 236223201337 2024-09-15T02:22:22.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:22:22.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:22:22.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:22:22.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-15T02:22:22.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:22:22.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:22:22.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:22:22.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:22:22.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:22:22.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:22:22.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:22:23.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-15T02:22:23.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:22:23.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:22:23.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:22:23.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-15T02:22:23.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:22:23.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:22:23.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-15T02:22:23.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:22:23.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:23.569 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-15T02:22:23.569 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:23.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-15T02:22:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:22:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:22:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:22:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:10' 2024-09-15T02:22:23.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-15T02:22:23.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-09-15T02:22:23.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:22:23.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:225: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-09-15T02:22:23.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2024-09-15T02:22:23.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:22:24.408 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:22:24.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-09-15T02:22:25.821 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile profile-jerasure does not exist 2024-09-15T02:22:25.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T02:22:25.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-09-15T02:22:25.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-09-15T02:22:25.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:153: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-09-15T02:22:25.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:154: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-09-15T02:22:25.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-09-15T02:22:26.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:161: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-15T02:22:26.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-15T02:22:27.888 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-lrc-a' already exists 2024-09-15T02:22:27.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:22:28.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:164: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T02:22:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T02:22:29.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:22:29.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:22:29.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T02:22:29.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-15T02:22:29.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-09-15T02:22:29.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:29.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-15T02:22:29.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:30.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-15T02:22:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T02:22:30.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2024-09-15T02:22:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-09-15T02:22:30.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T02:22:30.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-15T02:22:30.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-09-15T02:22:31.269 INFO:tasks.workunit.client.0.smithi130.stderr:osd.3 is already out. 2024-09-15T02:22:31.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-15T02:22:31.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<3\>' 2024-09-15T02:22:31.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-09-15T02:22:31.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:31.281 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-15T02:22:31.281 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:31.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-15T02:22:31.703 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:31.704 INFO:tasks.workunit.client.0.smithi130.stderr:7' 2024-09-15T02:22:31.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2024-09-15T02:22:31.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:22:33.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:22:33.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 3 2024-09-15T02:22:34.395 INFO:tasks.workunit.client.0.smithi130.stderr:osd.3 is already in. 2024-09-15T02:22:34.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:22:34.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:166: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-09-15T02:22:34.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2024-09-15T02:22:34.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-09-15T02:22:36.144 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-lrc-a' does not exist 2024-09-15T02:22:36.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:167: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-09-15T02:22:37.505 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile profile-lrc-a does not exist 2024-09-15T02:22:37.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T02:22:37.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-09-15T02:22:37.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-09-15T02:22:37.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:172: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-09-15T02:22:37.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:173: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-09-15T02:22:37.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-09-15T02:22:38.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-09-15T02:22:38.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-09-15T02:22:40.035 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-lrc' already exists 2024-09-15T02:22:40.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:22:41.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:182: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-09-15T02:22:41.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T02:22:41.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-09-15T02:22:41.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T02:22:41.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T02:22:41.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T02:22:41.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:22:42.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:22:42.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T02:22:42.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-15T02:22:42.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-09-15T02:22:42.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:42.604 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-15T02:22:42.604 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:42.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-09-15T02:22:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:10 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:7' 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-09-15T02:22:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-09-15T02:22:43.577 INFO:tasks.workunit.client.0.smithi130.stderr:osd.7 is already out. 2024-09-15T02:22:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-15T02:22:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-09-15T02:22:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<7\>' 2024-09-15T02:22:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:43.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-15T02:22:43.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:44.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-09-15T02:22:44.732 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:22:44.732 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:22:44.732 INFO:tasks.workunit.client.0.smithi130.stderr:10 2024-09-15T02:22:44.732 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:22:44.733 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:22:44.733 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:44.733 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-15T02:22:44.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2024-09-15T02:22:44.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:22:45.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:22:45.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 7 2024-09-15T02:22:46.421 INFO:tasks.workunit.client.0.smithi130.stderr:osd.7 is already in. 2024-09-15T02:22:46.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:22:46.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:184: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-09-15T02:22:46.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2024-09-15T02:22:46.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-09-15T02:22:48.384 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-lrc' does not exist 2024-09-15T02:22:48.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:185: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-09-15T02:22:49.826 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile profile-lrc does not exist 2024-09-15T02:22:49.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-15T02:22:49.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-09-15T02:22:49.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-09-15T02:22:49.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:232: TEST_rados_put_get_shec: local poolname=pool-shec 2024-09-15T02:22:49.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:233: TEST_rados_put_get_shec: local profile=profile-shec 2024-09-15T02:22:49.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:235: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-09-15T02:22:50.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-09-15T02:22:50.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-09-15T02:22:52.211 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-shec' already exists 2024-09-15T02:22:52.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:22:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:242: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-09-15T02:22:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-15T02:22:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-09-15T02:22:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-15T02:22:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-15T02:22:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-15T02:22:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-15T02:22:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-15T02:22:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-15T02:22:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-15T02:22:54.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:22:55.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:22:55.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-15T02:22:55.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-09-15T02:22:55.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-09-15T02:22:55.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:55.164 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-15T02:22:55.164 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:55.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-09-15T02:22:55.733 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:55.733 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2024-09-15T02:22:55.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2024-09-15T02:22:55.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-09-15T02:22:55.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-15T02:22:55.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-15T02:22:55.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-09-15T02:22:56.546 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. 2024-09-15T02:22:56.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-shec SOMETHING 2024-09-15T02:22:56.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-09-15T02:22:56.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<1\>' 2024-09-15T02:22:56.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-15T02:22:56.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-15T02:22:56.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:22:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-09-15T02:22:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:9 2024-09-15T02:22:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:22:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2024-09-15T02:22:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-15T02:22:59.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-15T02:22:59.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 1 2024-09-15T02:22:59.832 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already in. 2024-09-15T02:22:59.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-15T02:22:59.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:244: TEST_rados_put_get_shec: delete_pool pool-shec 2024-09-15T02:22:59.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2024-09-15T02:22:59.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-09-15T02:23:01.521 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-shec' does not exist 2024-09-15T02:23:01.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:245: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-09-15T02:23:02.770 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile profile-shec does not exist 2024-09-15T02:23:02.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-09-15T02:23:02.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2024-09-15T02:23:02.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-09-15T02:23:03.514 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'ecpool' does not exist 2024-09-15T02:23:03.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-09-15T02:23:03.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-15T02:23:03.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:23:03.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-15T02:23:03.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:23:03.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:23:03.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:23:03.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:23:03.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:23:03.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:23:03.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:23:03.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:23:03.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:23:03.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:23:03.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:23:03.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:23:03.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:23:03.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:23:03.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:23:03.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:23:03.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:23:03.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:23:03.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-15T02:23:03.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:23:03.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:03.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:23:03.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55401 2024-09-15T02:23:03.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:23:03.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:23:03.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-erasure-code 0 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:23:03.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:23:03.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:23:03.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:23:03.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:23:03.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:23:03.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:23:03.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:23:03.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:23:03.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:23:03.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:23:03.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:23:03.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:23:03.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T02:23:03.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-15T02:23:03.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:23:03.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:03.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.55401 2024-09-15T02:23:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.55401 2024-09-15T02:23:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:23:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:23:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T02:23:03.754 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T02:23:03.754 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T02:23:03.779 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-09-15T02:23:03.779 DEBUG:teuthology.orchestra.run.smithi130:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-09-15T02:23:03.841 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T02:23:03.847 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T02:23:03.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:23:03.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T02:23:03.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-erasure-eio 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none ' 2024-09-15T02:23:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-09-15T02:23:03.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-15T02:23:03.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-09-15T02:23:03.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_object_attr_read_error 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_recovery_multiple_errors 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_recovery_multiple_objects 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_recovery_multiple_objects_eio 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_recovery_unfound 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_single_recovery_error 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_get_bad_size_shard_0 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_get_bad_size_shard_1 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_get_subread_eio_shard_0 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_get_subread_eio_shard_1 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_get_subread_missing 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-09-15T02:23:03.851 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:23:03.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:23:03.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:23:03.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:23:03.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:23:03.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:23:03.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:23:03.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:23:03.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:23:03.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:23:03.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:23:03.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:23:03.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:23:03.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:23:03.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:23:03.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:23:03.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:23:03.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:23:03.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:23:03.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:23:03.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:23:04.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:23:04.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:23:04.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:23:04.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:23:04.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:23:04.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:23:04.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:23:04.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:23:04.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:23:04.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:23:04.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:23:04.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:23:04.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:23:04.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:23:04.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:23:04.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:23:04.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:23:04.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:23:05.402 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:23:05.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:23:05.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:23:05.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:23:05.677 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:23:05.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:23:06.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:23:06.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:23:06.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:23:06.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:23:06.689 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:06.689 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:06.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:23:06.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:23:06.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:23:06.755 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:23:06.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:23:06.764 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:23:04.211+0000 7ff5d1e16900 0 load: jerasure load: lrc 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2024-09-15T02:23:06.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-15T02:23:06.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:23:06.766 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-15T02:23:06.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-15T02:23:06.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:23:06.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:23:06.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:23:06.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:23:06.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:23:06.772 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 682f9635-1f1b-45ec-808f-4d33808fa0f1 2024-09-15T02:23:06.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=682f9635-1f1b-45ec-808f-4d33808fa0f1 2024-09-15T02:23:06.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 682f9635-1f1b-45ec-808f-4d33808fa0f1' 2024-09-15T02:23:06.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:23:06.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKReZmk6TcLhAA/BquZAtRHjZXwClmZcSxlw== 2024-09-15T02:23:06.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKReZmk6TcLhAA/BquZAtRHjZXwClmZcSxlw=="}' 2024-09-15T02:23:06.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 682f9635-1f1b-45ec-808f-4d33808fa0f1 -i td/test-erasure-eio/0/new.json 2024-09-15T02:23:07.084 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:23:07.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:23:07.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKReZmk6TcLhAA/BquZAtRHjZXwClmZcSxlw== --osd-uuid 682f9635-1f1b-45ec-808f-4d33808fa0f1 2024-09-15T02:23:07.123 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:07.120+0000 7f61bdf23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:07.125 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:07.123+0000 7f61bdf23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:07.126 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:07.124+0000 7f61bdf23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:07.127 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:07.124+0000 7f61bdf23440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:23:07.127 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:07.124+0000 7f61bdf23440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:23:07.127 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:07.124+0000 7f61bdf23440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:23:07.200 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:07.198+0000 7f61bdf23440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:23:13.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:23:13.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:23:13.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:23:13.004 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:23:13.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:23:13.487 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:23:13.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:23:13.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:23:13.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:23:13.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:23:13.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:23:13.514 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:13.511+0000 7fcb6500d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:13.515 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:13.513+0000 7fcb6500d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:13.516 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:13.514+0000 7fcb6500d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:13.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:23:13.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:23:13.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:23:13.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:23:13.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:23:13.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:13.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:23:13.863 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:23:13.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:13.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:14.032 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:14.030+0000 7fcb6500d440 -1 Falling back to public interface 2024-09-15T02:23:14.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:15.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:15.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:15.213 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:23:15.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:23:15.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:15.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:15.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:16.449 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:16.447+0000 7fcb6500d440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:23:16.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:16.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:16.565 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:23:16.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:23:16.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:16.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:17.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:18.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:18.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:18.032 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:23:18.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:23:18.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:18.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:18.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:19.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:19.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:19.454 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:23:19.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:23:19.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:19.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:19.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:20.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:20.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:20.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:23:20.843 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:23:20.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:20.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:21.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:22.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:22.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:22.290 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:23:22.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:23:22.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:22.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:22.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:23.678 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:23:23.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:23:23.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:23.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:24.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:25.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:25.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:25.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:23:25.169 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:23:25.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:25.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:25.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:26.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:26.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:26.572 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:23:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:23:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:26.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:26.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:27.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:27.935 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:23:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:23:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:28.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:29.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:29.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:29.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:23:29.685 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:23:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:30.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:31.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:31.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:31.449 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:23:31.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:23:31.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:31.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:32.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:33.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:33.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:33.487 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:23:33.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:23:33.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:33.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:34.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:35.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:35.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:35.253 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:23:35.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:23:35.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:35.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:35.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:36.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:36.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:36.982 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:23:36.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:23:36.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:36.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:37.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:38.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:38.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:38.794 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:23:38.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:23:38.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:38.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:23:39.205 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/89090,v1:127.0.0.1:6803/89090] [v2:127.0.0.1:6804/89090,v1:127.0.0.1:6805/89090] exists,up 682f9635-1f1b-45ec-808f-4d33808fa0f1 2024-09-15T02:23:39.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:23:39.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:23:39.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:23:39.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a67f9b73-cb3f-4a07-aa6f-e912a4f7b1f3 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a67f9b73-cb3f-4a07-aa6f-e912a4f7b1f3' 2024-09-15T02:23:39.213 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 a67f9b73-cb3f-4a07-aa6f-e912a4f7b1f3 2024-09-15T02:23:39.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:23:39.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArReZm0KdnDRAAbk7niU81oAFmg/Yf3D6J+w== 2024-09-15T02:23:39.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArReZm0KdnDRAAbk7niU81oAFmg/Yf3D6J+w=="}' 2024-09-15T02:23:39.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a67f9b73-cb3f-4a07-aa6f-e912a4f7b1f3 -i td/test-erasure-eio/1/new.json 2024-09-15T02:23:39.732 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:23:39.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:23:39.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArReZm0KdnDRAAbk7niU81oAFmg/Yf3D6J+w== --osd-uuid a67f9b73-cb3f-4a07-aa6f-e912a4f7b1f3 2024-09-15T02:23:39.772 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:39.771+0000 7fbd15323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:39.773 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:39.772+0000 7fbd15323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:39.775 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:39.773+0000 7fbd15323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:39.775 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:39.774+0000 7fbd15323440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:23:39.775 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:39.774+0000 7fbd15323440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:23:39.775 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:39.774+0000 7fbd15323440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:23:39.824 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:39.823+0000 7fbd15323440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:23:45.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:23:45.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:23:45.362 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:23:45.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:23:45.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:23:45.848 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:23:45.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:23:45.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:23:45.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:23:45.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:23:45.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:23:45.874 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:45.873+0000 7f37eaa51440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:45.876 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:45.875+0000 7f37eaa51440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:45.877 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:45.876+0000 7f37eaa51440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:23:46.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:23:46.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:23:46.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:23:46.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:23:46.199 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:23:46.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:23:46.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:46.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:23:46.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:46.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:46.390+0000 7f37eaa51440 -1 Falling back to public interface 2024-09-15T02:23:46.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:47.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:47.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:47.594 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:23:47.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:23:47.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:47.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:47.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:48.857 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:23:48.855+0000 7f37eaa51440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:23:48.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:48.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:48.946 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:23:48.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:23:48.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:48.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:49.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:50.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:50.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:50.535 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:23:50.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:23:50.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:50.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:50.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:51.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:51.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:51.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:23:51.947 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:23:51.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:51.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:52.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:53.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:53.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:53.344 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:23:53.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:23:53.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:53.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:53.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:54.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:54.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:54.758 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:23:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:23:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:54.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:55.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:56.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:56.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:56.125 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:23:56.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:23:56.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:56.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:56.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:57.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:57.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:57.665 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:23:57.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:23:57.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:57.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:58.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:23:59.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:23:59.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:23:59.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:23:59.060 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:23:59.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:23:59.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:23:59.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:00.425 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:24:00.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:24:00.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:00.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:24:01.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:02.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:02.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:02.163 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:24:02.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:24:02.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:02.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:24:02.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:03.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:03.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:03.982 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:24:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:24:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:03.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:24:04.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:05.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:05.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:05.709 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:24:05.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:24:05.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:05.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:24:06.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:07.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:07.797 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:24:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:24:07.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:07.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:24:08.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:09.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:09.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:09.525 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:24:09.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:24:09.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:09.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:24:10.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:11.352 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:24:11.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:11.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:11.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:24:11.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:11.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:24:11.762 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/90820,v1:127.0.0.1:6811/90820] [v2:127.0.0.1:6812/90820,v1:127.0.0.1:6813/90820] exists,up a67f9b73-cb3f-4a07-aa6f-e912a4f7b1f3 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:24:11.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:24:11.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:24:11.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5acc4f81-5321-4e01-80c8-2abbe6d94eb6 2024-09-15T02:24:11.770 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 5acc4f81-5321-4e01-80c8-2abbe6d94eb6 2024-09-15T02:24:11.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5acc4f81-5321-4e01-80c8-2abbe6d94eb6' 2024-09-15T02:24:11.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:24:11.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLReZm04SoLhAApYPAj/9XEsh4MOgcGQ0EqA== 2024-09-15T02:24:11.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLReZm04SoLhAApYPAj/9XEsh4MOgcGQ0EqA=="}' 2024-09-15T02:24:11.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5acc4f81-5321-4e01-80c8-2abbe6d94eb6 -i td/test-erasure-eio/2/new.json 2024-09-15T02:24:12.516 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:24:12.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:24:12.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLReZm04SoLhAApYPAj/9XEsh4MOgcGQ0EqA== --osd-uuid 5acc4f81-5321-4e01-80c8-2abbe6d94eb6 2024-09-15T02:24:12.556 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:12.554+0000 7f7488615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:12.558 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:12.556+0000 7f7488615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:12.559 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:12.557+0000 7f7488615440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:12.559 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:12.557+0000 7f7488615440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:24:12.559 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:12.557+0000 7f7488615440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:24:12.560 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:12.557+0000 7f7488615440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:24:12.631 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:12.630+0000 7f7488615440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:24:18.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:24:18.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:24:18.305 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:24:18.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:24:18.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:24:18.794 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:24:18.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:24:18.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:24:18.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:24:18.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:24:18.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:24:18.821 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:18.819+0000 7fe12f325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:18.822 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:18.821+0000 7fe12f325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:18.823 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:18.822+0000 7fe12f325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:19.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:24:19.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:24:19.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:24:19.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:24:19.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:24:19.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:19.170 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:24:19.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:24:19.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:19.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:19.342 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:19.341+0000 7fe12f325440 -1 Falling back to public interface 2024-09-15T02:24:19.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:20.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:20.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:20.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:24:20.523 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:24:20.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:20.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:20.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:21.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:21.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:21.883 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:24:21.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:24:21.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:21.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:21.899 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:21.898+0000 7fe12f325440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:24:22.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:23.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:24:23.359 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:24:23.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:23.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:23.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:24.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:24.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:24.789 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:24:24.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:24:24.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:24.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:25.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:26.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:26.270 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:24:26.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:26.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:24:26.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:26.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:26.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:27.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:27.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:27.694 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:24:27.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:24:27.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:27.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:28.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:29.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:29.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:29.152 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:24:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:24:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:29.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:30.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:30.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:30.661 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:24:30.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:24:30.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:30.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:31.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:32.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:32.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:32.051 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:24:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:24:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:32.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:33.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:33.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:33.432 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:24:33.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:24:33.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:33.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:34.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:35.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:35.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:35.054 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:24:35.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:24:35.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:35.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:35.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:36.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:36.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:36.970 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:24:36.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:24:36.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:36.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:38.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:39.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:39.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:39.048 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:24:39.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:24:39.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:39.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:39.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:40.541 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:24:40.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:24:40.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:40.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:41.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:42.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:42.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:42.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:24:42.313 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:24:42.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:42.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:43.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:44.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:44.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:44.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:24:44.007 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:24:44.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:44.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:24:44.539 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/92549,v1:127.0.0.1:6819/92549] [v2:127.0.0.1:6820/92549,v1:127.0.0.1:6821/92549] exists,up 5acc4f81-5321-4e01-80c8-2abbe6d94eb6 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:24:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:24:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:24:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:24:44.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:24:44.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:24:44.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ccb11fe-067e-41d3-bbe0-5efdafcc9a3d 2024-09-15T02:24:44.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0ccb11fe-067e-41d3-bbe0-5efdafcc9a3d' 2024-09-15T02:24:44.546 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 0ccb11fe-067e-41d3-bbe0-5efdafcc9a3d 2024-09-15T02:24:44.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:24:44.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsReZmDQ9SIRAAAseCTrN/0RQTqMzp0mPp+A== 2024-09-15T02:24:44.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsReZmDQ9SIRAAAseCTrN/0RQTqMzp0mPp+A=="}' 2024-09-15T02:24:44.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ccb11fe-067e-41d3-bbe0-5efdafcc9a3d -i td/test-erasure-eio/3/new.json 2024-09-15T02:24:45.398 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:24:45.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:24:45.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsReZmDQ9SIRAAAseCTrN/0RQTqMzp0mPp+A== --osd-uuid 0ccb11fe-067e-41d3-bbe0-5efdafcc9a3d 2024-09-15T02:24:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:45.436+0000 7fc5f6487440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:45.458 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:45.458+0000 7fc5f6487440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:45.460 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:45.459+0000 7fc5f6487440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:45.460 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:45.459+0000 7fc5f6487440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:24:45.460 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:45.459+0000 7fc5f6487440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:24:45.460 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:45.459+0000 7fc5f6487440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:24:45.498 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:45.498+0000 7fc5f6487440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:24:51.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:24:51.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:24:51.357 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:24:51.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:24:51.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:24:51.843 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:24:51.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:24:51.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:24:51.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:24:51.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:24:51.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:24:51.870 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:51.868+0000 7fbae2d80440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:51.889 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:51.888+0000 7fbae2d80440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:51.890 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:51.889+0000 7fbae2d80440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:24:52.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:24:52.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:24:52.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:24:52.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:24:52.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:24:52.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:52.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:24:52.218 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:24:52.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:52.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:24:52.412 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:52.411+0000 7fbae2d80440 -1 Falling back to public interface 2024-09-15T02:24:52.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:53.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:53.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:53.604 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:24:53.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:24:53.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:53.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:24:53.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:54.885 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:24:54.883+0000 7fbae2d80440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:24:54.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:54.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:54.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:24:54.966 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:24:54.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:54.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:24:55.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:56.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:56.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:56.546 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:24:56.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:24:56.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:56.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:24:57.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:58.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:58.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:58.144 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:24:58.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:24:58.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:58.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:24:58.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:24:59.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:24:59.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:24:59.622 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:24:59.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:24:59.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:24:59.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:00.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:01.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:01.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:01.135 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:25:01.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:25:01.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:01.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:01.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:02.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:02.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:02.526 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:25:02.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:25:02.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:02.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:02.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:03.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:03.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:03.922 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:25:03.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:25:03.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:03.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:04.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:05.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:05.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:05.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:25:05.345 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:25:05.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:05.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:05.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:06.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:06.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:06.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:25:06.861 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:25:06.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:06.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:07.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:08.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:08.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:08.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:25:08.291 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:25:08.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:08.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:09.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:10.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:10.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:10.345 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:25:10.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:25:10.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:10.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:11.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:12.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:12.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:12.487 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:25:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:25:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:12.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:13.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:14.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:14.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:14.259 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:25:14.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:25:14.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:14.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:15.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:16.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:16.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:16.071 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:25:16.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:25:16.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:16.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:16.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:17.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:17.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:17.873 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:25:17.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:25:17.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:17.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:18.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:19.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:19.813 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:25:19.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:25:19.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:19.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:25:20.232 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/94278,v1:127.0.0.1:6827/94278] [v2:127.0.0.1:6828/94278,v1:127.0.0.1:6829/94278] exists,up 0ccb11fe-067e-41d3-bbe0-5efdafcc9a3d 2024-09-15T02:25:20.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:25:20.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:25:20.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:25:20.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T02:25:20.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:25:20.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93a52ead-2245-4593-ad45-dcbe76ff5088 2024-09-15T02:25:20.239 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 93a52ead-2245-4593-ad45-dcbe76ff5088 2024-09-15T02:25:20.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 93a52ead-2245-4593-ad45-dcbe76ff5088' 2024-09-15T02:25:20.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:25:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQReZmFNwMDxAA+O5OEg4yl2Vla+ms64LXqg== 2024-09-15T02:25:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQReZmFNwMDxAA+O5OEg4yl2Vla+ms64LXqg=="}' 2024-09-15T02:25:20.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93a52ead-2245-4593-ad45-dcbe76ff5088 -i td/test-erasure-eio/4/new.json 2024-09-15T02:25:20.833 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:25:20.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T02:25:20.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQReZmFNwMDxAA+O5OEg4yl2Vla+ms64LXqg== --osd-uuid 93a52ead-2245-4593-ad45-dcbe76ff5088 2024-09-15T02:25:20.873 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:20.871+0000 7fd71ca5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:25:20.928 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:20.926+0000 7fd71ca5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:25:20.928 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:20.927+0000 7fd71ca5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:25:20.929 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:20.928+0000 7fd71ca5a440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:25:20.929 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:20.928+0000 7fd71ca5a440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:25:20.929 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:20.928+0000 7fd71ca5a440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:25:21.001 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:20.999+0000 7fd71ca5a440 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T02:25:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T02:25:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:25:27.579 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:25:27.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:25:27.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:25:28.062 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:25:28.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:25:28.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:25:28.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:25:28.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:25:28.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:25:28.088 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:28.087+0000 7f8aaec34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:25:28.090 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:28.089+0000 7f8aaec34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:25:28.091 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:28.090+0000 7f8aaec34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:25:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:25:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:25:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:25:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:25:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:25:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:28.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:25:28.418 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:25:28.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:28.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:28.605 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:28.604+0000 7f8aaec34440 -1 Falling back to public interface 2024-09-15T02:25:28.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:29.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:29.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:29.768 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:25:29.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:25:29.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:29.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:30.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:31.096 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:25:31.095+0000 7f8aaec34440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:25:31.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:31.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:31.121 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:25:31.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:25:31.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:31.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:31.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:32.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:32.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:32.521 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:25:32.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:25:32.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:32.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:33.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:34.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:34.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:34.026 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:25:34.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:25:34.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:34.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:34.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:35.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:35.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:35.592 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:25:35.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:25:35.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:35.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:36.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:37.058 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:25:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:25:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:37.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:37.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:38.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:38.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:38.455 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:25:38.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:25:38.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:38.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:38.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:39.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:39.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:39.955 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:25:39.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:25:39.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:39.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:40.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:41.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:41.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:41.346 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:25:41.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:25:41.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:41.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:41.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:42.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:42.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:42.780 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:25:42.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:25:42.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:42.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:43.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:44.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:44.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:44.222 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:25:44.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:25:44.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:44.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:44.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:45.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:45.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:45.789 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:25:45.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:25:45.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:45.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:46.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:47.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:47.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:47.495 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:25:47.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:25:47.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:47.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:48.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:49.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:49.223 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:25:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:25:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:49.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:50.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:51.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:51.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:51.017 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:25:51.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:25:51.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:51.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:51.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:52.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:52.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:52.562 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:25:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:25:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:53.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:25:54.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:25:54.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:25:54.315 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:25:54.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:25:54.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:25:54.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:25:54.735 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/96033,v1:127.0.0.1:6835/96033] [v2:127.0.0.1:6836/96033,v1:127.0.0.1:6837/96033] exists,up 93a52ead-2245-4593-ad45-dcbe76ff5088 2024-09-15T02:25:54.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:25:54.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:25:54.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:25:54.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:25:54.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:25:54.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:25:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:25:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:25:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:25:54.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:25:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:25:54.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:25:54.812 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:25:54.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:25:54.823 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:23:14.034+0000 7fcb6500d440 0 load: jerasure load: lrc 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:25:54.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T02:25:55.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:25:55.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:25:57.797 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:25:57.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:25:58.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:25:58.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:25:58.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:25:58.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:25:58.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:25:58.811 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:25:58.812 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:25:58.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:25:58.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:25:58.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:25:58.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:25:58.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:25:58.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:25:58.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:25:58.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:25:58.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:25:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:25:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:25:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:25:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:25:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-15T02:25:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:25:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:25:59.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:25:59.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803810 2024-09-15T02:25:59.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803810 2024-09-15T02:25:59.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810' 2024-09-15T02:25:59.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:25:59.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:25:59.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574881 2024-09-15T02:25:59.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574881 2024-09-15T02:25:59.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574881' 2024-09-15T02:25:59.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:25:59.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:25:59.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378644 2024-09-15T02:25:59.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378644 2024-09-15T02:25:59.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574881 2-81604378644' 2024-09-15T02:25:59.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:25:59.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:25:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084309 2024-09-15T02:25:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084309 2024-09-15T02:25:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574881 2-81604378644 3-120259084309' 2024-09-15T02:25:59.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:25:59.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:25:59.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724549 2024-09-15T02:25:59.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724549 2024-09-15T02:25:59.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574881 2-81604378644 3-120259084309 4-167503724549' 2024-09-15T02:25:59.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:25:59.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803810 2024-09-15T02:25:59.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:25:59.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:25:59.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803810 2024-09-15T02:25:59.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:25:59.837 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803810 2024-09-15T02:25:59.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803810 2024-09-15T02:25:59.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-09-15T02:25:59.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:26:00.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803810 2024-09-15T02:26:00.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:26:01.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:26:01.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:26:01.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803810 2024-09-15T02:26:01.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:26:01.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574881 2024-09-15T02:26:01.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:26:01.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:26:01.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574881 2024-09-15T02:26:01.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:26:01.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574881 2024-09-15T02:26:01.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-09-15T02:26:01.544 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574881 2024-09-15T02:26:01.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:26:01.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574881 -lt 55834574881 2024-09-15T02:26:01.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:26:01.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378644 2024-09-15T02:26:01.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:26:01.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:26:01.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378644 2024-09-15T02:26:01.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:26:01.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378644 2024-09-15T02:26:01.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-09-15T02:26:01.940 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378644 2024-09-15T02:26:01.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:26:02.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378644 -lt 81604378644 2024-09-15T02:26:02.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:26:02.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084309 2024-09-15T02:26:02.308 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:26:02.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:26:02.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084309 2024-09-15T02:26:02.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:26:02.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084309 2024-09-15T02:26:02.311 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084309 2024-09-15T02:26:02.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084309' 2024-09-15T02:26:02.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:26:02.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084310 -lt 120259084309 2024-09-15T02:26:02.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:26:02.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724549 2024-09-15T02:26:02.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:26:02.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:26:02.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724549 2024-09-15T02:26:02.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:26:02.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724549 2024-09-15T02:26:02.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724549' 2024-09-15T02:26:02.660 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 167503724549 2024-09-15T02:26:02.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:26:03.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724550 -lt 167503724549 2024-09-15T02:26:03.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:26:03.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:26:03.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:26:03.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:26:03.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:26:03.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:26:03.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:26:03.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:26:03.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:26:03.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:26:03.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:26:03.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:26:03.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:26:03.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:26:03.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:26:04.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:26:04.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:26:04.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:26:04.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:58.935500+0000 0'0 46:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-15T02:25:57.116058+0000 0'0 2024-09-15T02:25:57.116058+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:16:00.540669+0000 0 0 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:57.434298+0000 0'0 45:13 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:37:53.576192+0000 0 0 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:57.040399+0000 0'0 46:104 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:10:45.065238+0000 0 0 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:56.452973+0000 0'0 45:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T04:35:37.341113+0000 0 0 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:24:45.062725+0000 0'0 46:43 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:23:58.797151+0000 0 0 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-15T02:26:04.631 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T02:26:04.632 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-15T02:26:04.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:26:04.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:26:04.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:26:04.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T02:26:04.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:26:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:04.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-09-15T02:26:04.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-09-15T02:26:04.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-15T02:26:04.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:26:04.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T02:26:05.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2024-09-15T02:26:05.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-09-15T02:26:05.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:26:05.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:26:05.162 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:26:05.162 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:26:05.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:26:05.514 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:26:05.514 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:26:05.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:26:05.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:26:05.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2024-09-15T02:26:06.691 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already down. 2024-09-15T02:26:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2024-09-15T02:26:07.313 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2024-09-15T02:26:07.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T02:26:07.742 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-09-15T02:25:58.935500+0000 47'1 47:13 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-15T02:25:57.116058+0000 0'0 2024-09-15T02:25:57.116058+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:16:00.540669+0000 0 0 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:57.434298+0000 0'0 46:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:37:53.576192+0000 0 0 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:57.040399+0000 0'0 46:104 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:10:45.065238+0000 0 0 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:56.452973+0000 0'0 46:15 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T04:35:37.341113+0000 0 0 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-15T02:26:06.789850+0000 0'0 50:53 [1,0] 1 [1,0] 1 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:33:48.239450+0000 0 0 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T02:26:07.743 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-15T02:26:07.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-15T02:26:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:4+0 records in 2024-09-15T02:26:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:4+0 records out 2024-09-15T02:26:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.4393e-05 s, 43.4 MB/s 2024-09-15T02:26:07.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2024-09-15T02:26:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:08.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:08.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:08.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:08.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:09.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:09.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:09.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:09.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:09.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:09.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:09.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:09.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:09.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:09.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:10.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:10.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:10.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:10.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:10.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:10.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:11.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:11.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:11.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:11.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:11.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:11.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:11.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:11.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:12.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:12.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:12.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:12.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:12.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:12.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:12.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:12.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:13.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:13.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:13.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:13.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:13.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:13.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:14.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:14.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:14.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:14.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:14.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:14.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:14.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:14.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:14.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:14.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:15.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:15.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:15.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:15.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:15.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:15.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:15.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:15.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:16.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:16.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:16.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:16.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:16.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:16.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:16.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:16.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:17.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:17.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:17.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:17.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:17.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:17.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:17.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:17.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:18.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:18.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:18.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:18.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:18.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:18.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:18.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:18.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:19.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:19.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:19.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:19.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:19.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:19.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:19.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:19.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:20.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:20.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:20.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:20.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:20.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:20.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:20.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:20.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:21.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:21.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:21.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:21.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:21.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:21.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:21.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:21.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:22.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:22.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:22.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:22.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:22.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:22.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:22.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:22.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:23.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:23.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:23.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:23.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:23.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:23.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:23.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:23.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:24.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:24.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:24.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:24.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:24.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:24.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:24.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:24.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:25.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:25.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:25.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:25.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:25.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:25.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:25.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:25.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:26.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:26.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:26.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:26.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:26.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:26.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:27.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:27.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:27.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:27.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:27.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:27.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:27.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:27.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:27.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:28.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:28.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:28.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:28.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:28.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:28.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:28.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:28.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:29.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:29.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:29.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:29.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:29.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:30.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:30.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:30.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:30.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:30.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:30.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:31.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:31.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:31.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:31.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:31.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:31.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:31.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:31.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:32.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:32.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:32.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:32.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:32.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:32.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:32.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:32.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:32.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:32.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:33.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:33.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:33.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:33.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:33.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:34.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:34.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:34.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:34.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:34.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:34.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:34.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:35.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:35.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:35.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:35.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:35.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:35.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:35.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:35.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:35.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:36.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:36.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:36.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:36.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:36.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:36.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:36.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:36.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:37.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:37.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:37.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:37.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:37.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:37.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:37.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:37.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:38.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:38.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:38.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:38.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:38.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:38.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:38.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:38.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:39.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:39.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:39.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:39.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:39.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:39.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:40.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:40.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:40.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:40.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:40.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:40.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:40.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:41.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:41.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:41.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:41.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:42.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:42.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:42.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:42.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:42.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:42.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:42.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:42.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:43.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:43.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:43.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:43.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:43.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:43.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:43.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:43.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:44.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:44.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:44.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:44.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:44.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:44.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:45.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:45.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:45.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:45.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:45.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:45.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:45.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:45.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:46.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:46.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:46.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:46.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:46.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:46.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:46.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:46.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:47.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:47.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:47.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:47.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:47.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:47.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:47.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:47.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:48.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:48.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:48.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:48.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:48.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:48.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:48.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:48.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:49.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:49.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:49.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:49.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:49.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:49.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:49.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:49.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:50.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:50.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:50.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:50.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:50.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:50.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:50.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:50.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:51.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:51.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:51.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:51.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:51.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:51.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:51.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:51.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:52.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:52.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:52.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:52.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:52.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:52.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:52.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:52.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:53.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:53.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:53.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:53.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:53.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:53.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:53.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:53.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:54.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:54.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:54.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:54.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:54.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:54.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:54.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:54.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:55.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:55.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:55.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:55.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:55.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:55.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:55.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:55.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:56.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:56.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:56.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:56.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:56.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:56.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:56.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:56.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:57.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:57.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:57.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:57.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:57.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:57.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:57.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:57.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:58.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:58.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:58.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:58.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:58.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:58.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:58.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:58.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:59.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:59.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:59.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:59.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:59.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:59.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-09-15T02:26:59.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:26:59.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:00.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:00.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:00.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:00.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:00.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:00.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:00.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:00.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:01.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:01.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:01.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:01.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:01.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:01.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:01.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:01.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:02.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:02.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:02.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:02.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:02.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:02.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:02.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:02.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:03.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:03.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:03.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:03.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:03.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:03.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:03.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:03.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:04.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:04.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:04.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:04.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:04.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:04.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:04.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:04.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:05.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:05.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:05.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:05.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:05.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:05.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:05.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:05.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:06.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:06.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:06.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:06.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:06.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:06.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:06.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:06.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:07.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:07.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:07.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:07.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:07.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:07.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:07.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:07.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:08.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:08.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:08.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:08.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:08.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:08.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:08.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:08.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:08.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:08.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:09.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:09.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:09.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:09.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:09.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:09.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:09.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:09.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:10.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:10.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:10.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:10.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:10.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:10.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:10.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:10.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:11.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:11.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:11.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:11.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:11.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:11.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:11.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:12.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:12.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:12.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:12.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:12.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:12.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:13.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:13.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:13.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:13.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:13.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:13.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:13.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:13.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:14.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:14.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:14.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:14.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:14.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:14.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:14.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:14.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:15.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:15.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:15.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:15.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:15.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:15.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:16.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:16.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:16.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:16.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:16.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:16.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:16.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:16.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:17.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:17.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:17.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:17.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:17.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:17.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:17.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:17.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:18.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:18.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:18.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:18.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:18.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:18.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:18.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:18.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:19.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:19.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:19.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:19.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:19.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:19.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:19.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:19.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:20.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:20.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:20.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:20.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:20.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:20.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:20.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:21.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:21.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:21.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:21.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:21.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:21.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:21.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:21.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:22.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:22.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:22.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:22.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:22.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:22.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:22.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:22.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:23.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:23.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:23.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:27:23.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-09-15T02:27:23.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-09-15T02:27:23.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:27:23.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:27:23.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:27:23.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:27:23.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:27:23.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:27:23.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-15T02:27:23.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:27:23.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:27:23.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:27:23.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T02:27:23.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:27:23.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj250 2024-09-15T02:27:23.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:27:23.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-15T02:27:23.543 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-15T02:27:23.543 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr:2147483647' 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T02:27:23.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:27:23.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T02:27:23.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:27:23.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T02:27:23.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:27:23.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:27:23.906 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:27:23.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:27:23.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:27:23.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:27:23.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:27:23.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T02:27:23.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:27:23.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:27:23.985 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:27:23.985 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:27:23.986 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:27:23.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:27:23.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:27:23.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-09-15T02:27:24.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-09-15T02:27:24.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:27:24.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:27:24.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj250 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:27:24.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-15T02:27:24.070 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-15T02:27:24.070 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:27:24.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:27:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:27:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:27:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:27:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:2147483647' 2024-09-15T02:27:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-15T02:27:24.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T02:27:24.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:27:24.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T02:27:24.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:27:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T02:27:24.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T02:27:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T02:27:24.466 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T02:27:24.466 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:27:24.466 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:27:24.466 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:27:24.466 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T02:27:24.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:27:24.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:27:24.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:27:24.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:27:24.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T02:27:24.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T02:27:24.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T02:27:24.543 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:27:24.543 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:27:24.543 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:27:24.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T02:27:24.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:27:24.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:27:24.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:27:24.629 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:27:24.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:27:24.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:27:24.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T02:27:24.631 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:27:24.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:27:24.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T02:27:24.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T02:27:24.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:27:24.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:27:24.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:27:24.649 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:27:24.648+0000 7f2f9960a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:27:24.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:27:24.650+0000 7f2f9960a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:27:24.652 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:27:24.651+0000 7f2f9960a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:27:25.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T02:27:25.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:27:25.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:27:25.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:27:25.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:27:25.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:27:25.020 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:27:25.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:27:25.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:27:25.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:27:25.170 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:27:25.169+0000 7f2f9960a440 -1 Falling back to public interface 2024-09-15T02:27:25.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:27:26.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:27:26.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:27:26.413 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:27:26.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:27:26.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:27:26.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:27:26.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:27:27.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:27:27.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:27:27.774 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:27:27.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:27:27.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:27:27.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:27:28.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:27:28.151 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:27:28.150+0000 7f2f9960a440 -1 osd.2 47 log_to_monitors true 2024-09-15T02:27:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:27:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:27:29.131 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:27:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:27:29.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:27:29.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:27:29.619 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up out weight 0 up_from 55 up_thru 19 down_at 48 last_clean_interval [19,47) [v2:127.0.0.1:6818/103698,v1:127.0.0.1:6819/103698] [v2:127.0.0.1:6820/103698,v1:127.0.0.1:6821/103698] exists,up 5acc4f81-5321-4e01-80c8-2abbe6d94eb6 2024-09-15T02:27:29.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:27:29.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:27:29.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:27:29.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2024-09-15T02:27:30.758 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2024-09-15T02:27:30.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2024-09-15T02:27:45.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2024-09-15T02:27:45.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:45.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:45.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:45.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:45.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:45.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:46.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:46.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:46.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:46.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:46.127 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:46.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:46.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:47.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:47.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:47.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:47.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:47.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:47.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:47.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:47.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:47.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:47.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:47.483 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:48.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:48.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:48.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:48.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:48.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:48.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:48.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:48.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:48.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:48.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:48.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:48.833 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:48.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:49.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:49.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:49.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:49.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:49.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:49.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:50.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:50.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:50.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:50.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:50.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:50.187 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:50.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:51.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:51.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:51.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:51.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:51.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:51.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:51.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:51.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:51.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:51.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:51.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:51.542 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:51.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:52.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:52.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:52.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:52.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:52.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:52.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:52.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:52.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:52.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:52.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:52.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:52.926 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:52.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:53.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:53.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:53.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:53.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:53.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:53.928 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:54.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:54.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:54.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:54.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:54.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:54.277 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:54.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:55.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:55.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:55.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:55.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:55.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:55.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:55.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:55.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:55.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:55.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:55.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:55.631 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:55.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:56.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:56.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:56.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:56.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:56.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:56.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:56.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:56.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:56.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:56.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T02:27:56.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:56.981 INFO:tasks.workunit.client.0.smithi130.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T02:27:56.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-15T02:27:57.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-15T02:27:57.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-15T02:27:57.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:27:57.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:27:57.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:27:57.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:27:58.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-15T02:27:58.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-15T02:27:58.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-15T02:27:58.330 INFO:tasks.workunit.client.0.smithi130.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-15T02:27:58.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-09-15T02:27:58.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2024-09-15T02:27:58.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T02:27:58.662 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T02:27:58.662 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-15T02:27:54.250604+0000 53'301 62:904 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-09-15T02:25:57.116058+0000 0'0 2024-09-15T02:25:57.116058+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:57:17.024739+0000 0 0 2024-09-15T02:27:58.662 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:27:31.546258+0000 0'0 61:55 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T02:31:54.605893+0000 0 0 2024-09-15T02:27:58.662 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:57.040399+0000 0'0 61:116 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:10:45.065238+0000 0 0 2024-09-15T02:27:58.663 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:56.452973+0000 0'0 61:29 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T04:35:37.341113+0000 0 0 2024-09-15T02:27:58.663 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:27:31.546383+0000 0'0 61:84 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T03:05:35.372060+0000 0 0 2024-09-15T02:27:58.663 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-15T02:27:58.663 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T02:27:58.663 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-15T02:27:58.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T02:27:58.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:27:58.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:27:58.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:27:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:27:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:27:58.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:27:58.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2024-09-15T02:28:03.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-15T02:28:04.317 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T02:28:04.317 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-09-15T02:28:00.112634+0000 53'301 65:919 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-15T02:25:57.116058+0000 0'0 2024-09-15T02:25:57.116058+0000 0 0 periodic scrub scheduled @ 2024-09-16T09:02:20.586805+0000 0 0 2024-09-15T02:28:04.317 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-15T02:27:59.533887+0000 0'0 64:65 [4,1] 4 [4,1] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:52:31.895887+0000 0 0 2024-09-15T02:28:04.317 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:57.040399+0000 0'0 61:116 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:10:45.065238+0000 0 0 2024-09-15T02:28:04.317 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:25:56.452973+0000 0'0 64:32 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T04:35:37.341113+0000 0 0 2024-09-15T02:28:04.317 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-15T02:28:00.020573+0000 0'0 65:94 [1,0] 1 [1,0] 1 0'0 2024-09-15T02:23:05.565431+0000 0'0 2024-09-15T02:23:05.565431+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:34:43.799871+0000 0 0 2024-09-15T02:28:04.317 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-15T02:28:04.318 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T02:28:04.318 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-15T02:28:04.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:28:04.428 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-15T02:28:04.428 INFO:tasks.workunit.client.0.smithi130.stdout: "num_missing": 1, 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "num_unfound": 1, 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": [ 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "oid": { 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "oid": "obj250", 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "key": "", 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "snapid": -2, 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": 2249616407, 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "max": 0, 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 2, 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "namespace": "" 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "need": "53'251", 2024-09-15T02:28:04.429 INFO:tasks.workunit.client.0.smithi130.stdout: "have": "0'0", 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "none", 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "locations": [ 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "NotRecovering", 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "available_might_have_unfound": true, 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "might_have_unfound": [ 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.430 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "2(4)", 2024-09-15T02:28:04.431 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "osd is down" 2024-09-15T02:28:04.431 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.431 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.431 INFO:tasks.workunit.client.0.smithi130.stdout: "more": false 2024-09-15T02:28:04.431 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-15T02:28:04.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-09-15T02:28:04.539 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-15T02:28:04.539 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trimq": "[]", 2024-09-15T02:28:04.539 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trimq_len": 0, 2024-09-15T02:28:04.539 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-15T02:28:04.539 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 65, 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.540 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_recovery_backfill": [ 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "info": { 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s0", 2024-09-15T02:28:04.541 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'301", 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'250", 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "53'200", 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 301, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 45, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 45, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 63, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 45, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:28:04.542 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 63, 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 63, 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 45, 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'301", 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 920, 2024-09-15T02:28:04.543 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 65, 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:28:00.112634+0000", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:28:00.112634+0000", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:28:00.112634+0000", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:28:00.112634+0000", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:26:04.802783+0000", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:28:00.079315+0000", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:28:00.079315+0000", 2024-09-15T02:28:04.544 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:28:00.112634+0000", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:27:59.793529+0000", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:27:59.793308+0000", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 63, 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "53'200", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "53'200", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 45, 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.545 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 200, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:28:04.546 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T09:02:20.586805+0000", 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 1232896, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 301, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 1505, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 1, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 1, 2024-09-15T02:28:04.547 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 303, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 1, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 301, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 301, 2024-09-15T02:28:04.548 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 1204, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:28:04.549 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:28:04.550 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:28:04.551 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.552 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.553 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 300 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.554 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),4(2)", 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 1 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:28:04.555 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:28:04.556 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: "peer_info": [ 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: "peer": "0(3)", 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s3", 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'301", 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'301", 2024-09-15T02:28:04.557 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "53'200", 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 301, 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 45, 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 45, 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 63, 2024-09-15T02:28:04.558 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 45, 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 63, 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 63, 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 45, 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.559 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'301", 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 909, 2024-09-15T02:28:04.560 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 63, 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:26:04.802783+0000", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.561 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:26:06.588162+0000", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:26:06.588051+0000", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 63, 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "53'200", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "53'200", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 45, 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.562 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 200, 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:28:04.563 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T04:18:10.370769+0000", 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:28:04.564 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 1232896, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 301, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 1505, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 301, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 301, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 301, 2024-09-15T02:28:04.565 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 1204, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:28:04.566 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:28:04.567 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.568 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 301 2024-09-15T02:28:04.569 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:28:04.570 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "peer": "1(1)", 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s1", 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'301", 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'301", 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "53'200", 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 301, 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:28:04.571 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 45, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 45, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 63, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 45, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 63, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 63, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 45, 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.572 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'301", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 329, 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 53, 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.573 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:26:04.802783+0000", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:26:06.588162+0000", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:26:06.588051+0000", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 63, 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "53'200", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "53'200", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 45, 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.574 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 200, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:28:04.575 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T04:18:10.370769+0000", 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 1232896, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 301, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 1505, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 1, 2024-09-15T02:28:04.576 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 301, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 301, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 301, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 1204, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:28:04.577 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:28:04.578 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.579 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:28:04.580 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 301 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.581 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "peer": "4(2)", 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s2", 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'301", 2024-09-15T02:28:04.582 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'301", 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "53'200", 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 301, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 45, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 45, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 63, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 45, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:28:04.583 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 63, 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 63, 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 45, 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:28:04.584 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'301", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 329, 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 53, 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:26:04.802783+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:26:06.924437+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:27:23.535662+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:26:06.588162+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:26:06.588051+0000", 2024-09-15T02:28:04.585 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 63, 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "53'200", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "53'200", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 45, 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 46, 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:25:57.116058+0000", 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:28:04.586 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 200, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:28:04.587 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T04:18:10.370769+0000", 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 1232896, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 301, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 1505, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 301, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:28:04.588 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 301, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 301, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 1204, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:28:04.589 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:28:04.590 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.591 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: 2147483647 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:28:04.592 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 301 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:28:04.593 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 65, 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_state": [ 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "Started/Primary/Active", 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "enter_time": "2024-09-15T02:27:59.793331+0000", 2024-09-15T02:28:04.594 INFO:tasks.workunit.client.0.smithi130.stdout: "might_have_unfound": [ 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "0(3)", 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "already probed" 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "1(1)", 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "already probed" 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "2(4)", 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "osd is down" 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "4(2)", 2024-09-15T02:28:04.595 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "already probed" 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_progress": { 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "backfill_targets": [], 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "waiting_on_backfill": [], 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill_started": "MIN", 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "backfill_info": { 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "begin": "MIN", 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "end": "MIN", 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": [] 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "peer_backfill_info": [], 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "backfills_in_flight": [], 2024-09-15T02:28:04.596 INFO:tasks.workunit.client.0.smithi130.stdout: "recovering": [], 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_backend": { 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_ops": [], 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: "read_ops": [] 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "Started", 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: "enter_time": "2024-09-15T02:27:59.030947+0000" 2024-09-15T02:28:04.597 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "scrubber": { 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "active": false, 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "must_scrub": false, 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "must_deep_scrub": false, 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "must_repair": false, 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "need_auto": false, 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_reg_stamp": "2024-09-16T09:02:20.586805+0000", 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "schedule": "scrub scheduled @ 2024-09-16T09:02:20.586805+0000" 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout: "agent_state": {} 2024-09-15T02:28:04.598 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-15T02:28:04.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:28:04.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2024-09-15T02:28:04.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:28:04.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-09-15T02:28:04.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2024-09-15T02:28:04.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2024-09-15T02:28:04.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:28:04.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-09-15T02:28:04.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-09-15T02:28:04.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2024-09-15T02:28:04.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-09-15T02:28:04.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:28:04.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-09-15T02:28:04.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-09-15T02:28:04.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2024-09-15T02:28:04.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-09-15T02:28:04.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-15T02:28:04.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:28:04.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:28:04.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T02:28:04.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:28:04.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:28:04.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:28:04.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:28:04.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:28:04.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:28:04.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:28:04.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:28:04.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:28:04.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:28:04.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:28:04.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:28:04.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:28:04.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:28:04.996 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:28:04.996 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:04.996 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:04.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:28:04.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:28:04.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:28:04.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:28:04.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:28:04.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:28:04.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:28:04.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:28:04.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:28:04.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:28:04.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:28:04.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:28:04.999 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:28:05.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T02:28:05.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:28:05.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T02:28:05.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T02:28:05.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:28:05.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:28:05.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:28:05.010 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:05.009+0000 7f9775614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:05.012 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:05.011+0000 7f9775614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:05.013 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:05.012+0000 7f9775614440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T02:28:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:28:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:28:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:28:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:28:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:28:05.381 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:28:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:05.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:28:05.542 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:05.541+0000 7f9775614440 -1 Falling back to public interface 2024-09-15T02:28:05.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:06.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:06.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:06.731 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:28:06.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:28:06.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:06.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:28:07.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:08.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:08.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:08.090 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:28:08.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:28:08.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:08.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:28:08.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:08.954 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:08.953+0000 7f9775614440 -1 osd.2 62 log_to_monitors true 2024-09-15T02:28:09.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:09.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:09.438 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:28:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:28:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:09.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:28:09.989 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 67 up_thru 19 down_at 63 last_clean_interval [55,62) [v2:127.0.0.1:6818/104782,v1:127.0.0.1:6819/104782] [v2:127.0.0.1:6820/104782,v1:127.0.0.1:6821/104782] exists,up 5acc4f81-5321-4e01-80c8-2abbe6d94eb6 2024-09-15T02:28:09.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:28:09.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:28:09.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:28:09.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-15T02:28:14.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2024-09-15T02:28:14.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-15T02:28:15.234 INFO:tasks.workunit.client.0.smithi130.stderr:pg has no unfound objects 2024-09-15T02:28:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2024-09-15T02:28:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:28:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:28:15.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:28:15.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:28:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:28:15.245 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:28:15.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:28:15.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:28:15.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:28:15.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:28:15.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:28:15.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:28:15.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:28:15.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:28:15.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:28:15.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:28:15.688 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:28:15.688 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:28:15.688 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:28:15.688 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-15T02:28:15.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:28:15.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:28:15.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:28:15.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803839 2024-09-15T02:28:15.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803839 2024-09-15T02:28:15.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839' 2024-09-15T02:28:15.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:28:15.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:28:15.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574910 2024-09-15T02:28:15.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574910 2024-09-15T02:28:15.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839 1-55834574910' 2024-09-15T02:28:15.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:28:15.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:28:16.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=287762808836 2024-09-15T02:28:16.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 287762808836 2024-09-15T02:28:16.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839 1-55834574910 2-287762808836' 2024-09-15T02:28:16.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:28:16.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:28:16.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084339 2024-09-15T02:28:16.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084339 2024-09-15T02:28:16.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839 1-55834574910 2-287762808836 3-120259084339' 2024-09-15T02:28:16.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:28:16.204 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:28:16.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724579 2024-09-15T02:28:16.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724579 2024-09-15T02:28:16.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839 1-55834574910 2-287762808836 3-120259084339 4-167503724579' 2024-09-15T02:28:16.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:28:16.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803839 2024-09-15T02:28:16.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:28:16.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:28:16.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803839 2024-09-15T02:28:16.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:28:16.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803839 2024-09-15T02:28:16.312 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803839 2024-09-15T02:28:16.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803839' 2024-09-15T02:28:16.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:28:16.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803837 -lt 25769803839 2024-09-15T02:28:16.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:28:17.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:28:17.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:28:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803839 -lt 25769803839 2024-09-15T02:28:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:28:18.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574910 2024-09-15T02:28:18.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:28:18.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:28:18.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574910 2024-09-15T02:28:18.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:28:18.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574910 2024-09-15T02:28:18.091 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574910 2024-09-15T02:28:18.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574910' 2024-09-15T02:28:18.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:28:18.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574910 -lt 55834574910 2024-09-15T02:28:18.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:28:18.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-287762808836 2024-09-15T02:28:18.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:28:18.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:28:18.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-287762808836 2024-09-15T02:28:18.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:28:18.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=287762808836 2024-09-15T02:28:18.851 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 287762808836 2024-09-15T02:28:18.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 287762808836' 2024-09-15T02:28:18.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:28:19.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 287762808836 -lt 287762808836 2024-09-15T02:28:19.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:28:19.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084339 2024-09-15T02:28:19.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:28:19.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:28:19.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084339 2024-09-15T02:28:19.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:28:19.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084339 2024-09-15T02:28:19.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084339' 2024-09-15T02:28:19.353 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084339 2024-09-15T02:28:19.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:28:19.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084339 -lt 120259084339 2024-09-15T02:28:19.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:28:19.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724579 2024-09-15T02:28:19.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:28:19.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:28:19.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724579 2024-09-15T02:28:19.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:28:19.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724579 2024-09-15T02:28:19.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724579' 2024-09-15T02:28:19.828 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 167503724579 2024-09-15T02:28:19.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:28:20.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724579 -lt 167503724579 2024-09-15T02:28:20.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:28:20.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:28:20.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:28:20.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:28:20.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:28:20.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:28:20.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:28:20.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:28:20.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:28:20.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:28:20.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:28:21.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T02:28:21.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:28:21.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:28:21.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:28:21.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-15T02:28:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-15T02:28:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T02:28:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-15T02:28:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:28:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:28:21.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:28:21.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:28:21.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:28:21.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:28:21.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:28:21.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:28:21.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:28:21.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:28:21.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:28:21.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:28:21.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:28:21.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:28:22.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:28:22.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:28:22.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:28:22.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2024-09-15T02:28:22.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-15T02:28:22.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-15T02:28:22.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-15T02:28:22.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-15T02:28:22.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-15T02:28:22.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-15T02:28:22.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-15T02:28:22.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-15T02:28:22.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-15T02:28:22.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-15T02:28:22.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-15T02:28:22.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-15T02:28:22.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-15T02:28:22.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-15T02:28:22.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-15T02:28:22.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-15T02:28:22.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-15T02:28:22.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-15T02:28:22.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-15T02:28:22.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:22.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:22.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-15T02:28:22.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-15T02:28:23.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-15T02:28:23.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-15T02:28:23.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-15T02:28:23.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-15T02:28:23.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-15T02:28:23.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-15T02:28:23.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-15T02:28:23.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-15T02:28:23.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-15T02:28:23.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-15T02:28:23.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-15T02:28:23.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-15T02:28:23.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-15T02:28:23.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-15T02:28:23.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-15T02:28:23.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-15T02:28:23.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-15T02:28:23.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-15T02:28:23.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-15T02:28:23.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-15T02:28:23.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-15T02:28:23.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-15T02:28:23.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-15T02:28:23.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-15T02:28:23.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-15T02:28:23.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-15T02:28:23.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-15T02:28:23.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-15T02:28:23.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-15T02:28:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-15T02:28:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-15T02:28:23.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-15T02:28:23.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-15T02:28:23.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-15T02:28:23.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:23.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:23.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-15T02:28:23.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-15T02:28:24.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-15T02:28:24.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-15T02:28:24.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-15T02:28:24.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-15T02:28:24.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-15T02:28:24.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-15T02:28:24.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-15T02:28:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-15T02:28:24.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-15T02:28:24.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-15T02:28:24.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-15T02:28:24.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-15T02:28:24.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-15T02:28:24.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-15T02:28:24.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-15T02:28:24.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-15T02:28:24.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-15T02:28:24.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-15T02:28:24.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-15T02:28:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-15T02:28:24.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-15T02:28:24.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-15T02:28:24.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-15T02:28:24.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-15T02:28:24.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-15T02:28:24.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-15T02:28:24.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-15T02:28:24.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-15T02:28:24.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-15T02:28:24.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-15T02:28:24.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-15T02:28:24.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-15T02:28:24.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-15T02:28:24.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-15T02:28:24.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-15T02:28:24.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-15T02:28:24.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-15T02:28:24.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:24.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:24.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-15T02:28:24.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-15T02:28:25.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-15T02:28:25.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-15T02:28:25.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-15T02:28:25.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-15T02:28:25.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-15T02:28:25.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-15T02:28:25.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-15T02:28:25.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-15T02:28:25.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-15T02:28:25.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-15T02:28:25.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-15T02:28:25.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-15T02:28:25.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-15T02:28:25.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-15T02:28:25.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-15T02:28:25.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-15T02:28:25.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-15T02:28:25.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-15T02:28:25.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-15T02:28:25.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-15T02:28:25.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-15T02:28:25.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-15T02:28:25.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-15T02:28:25.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-15T02:28:25.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-15T02:28:25.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-15T02:28:25.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-15T02:28:25.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-15T02:28:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-15T02:28:25.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-15T02:28:25.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-15T02:28:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-15T02:28:25.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-15T02:28:25.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-15T02:28:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-09-15T02:28:25.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:25.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:25.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-15T02:28:25.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-15T02:28:26.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-15T02:28:26.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-15T02:28:26.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-15T02:28:26.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-15T02:28:26.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-15T02:28:26.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-15T02:28:26.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-15T02:28:26.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-15T02:28:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-15T02:28:26.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-15T02:28:26.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-15T02:28:26.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-15T02:28:26.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-15T02:28:26.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-15T02:28:26.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-15T02:28:26.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-15T02:28:26.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-15T02:28:26.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-15T02:28:26.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-15T02:28:26.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-15T02:28:26.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-15T02:28:26.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-15T02:28:26.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-15T02:28:26.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-15T02:28:26.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-15T02:28:26.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-15T02:28:26.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-15T02:28:26.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-15T02:28:26.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-15T02:28:26.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-15T02:28:26.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-15T02:28:26.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-15T02:28:26.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-15T02:28:26.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-15T02:28:26.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-15T02:28:26.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-15T02:28:26.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-09-15T02:28:26.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:26.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:26.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-15T02:28:26.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-15T02:28:27.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-15T02:28:27.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-15T02:28:27.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-15T02:28:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-15T02:28:27.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-15T02:28:27.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-15T02:28:27.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-15T02:28:27.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-15T02:28:27.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-15T02:28:27.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-15T02:28:27.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-15T02:28:27.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-15T02:28:27.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-15T02:28:27.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-15T02:28:27.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-15T02:28:27.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-15T02:28:27.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-15T02:28:27.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-15T02:28:27.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-15T02:28:27.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-15T02:28:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-15T02:28:27.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-15T02:28:27.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-15T02:28:27.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-15T02:28:27.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-15T02:28:27.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-15T02:28:27.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-15T02:28:27.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-15T02:28:27.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-15T02:28:27.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-15T02:28:27.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-15T02:28:27.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-15T02:28:27.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-09-15T02:28:27.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:27.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:27.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-15T02:28:27.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-15T02:28:28.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-15T02:28:28.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-15T02:28:28.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-15T02:28:28.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-15T02:28:28.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-15T02:28:28.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-15T02:28:28.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-15T02:28:28.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-15T02:28:28.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-15T02:28:28.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-15T02:28:28.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-15T02:28:28.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-15T02:28:28.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-15T02:28:28.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-15T02:28:28.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-15T02:28:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-15T02:28:28.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-15T02:28:28.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-15T02:28:28.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-15T02:28:28.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-15T02:28:28.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-15T02:28:28.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-15T02:28:28.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-15T02:28:28.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-15T02:28:28.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-15T02:28:28.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-15T02:28:28.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-15T02:28:28.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-15T02:28:28.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-15T02:28:28.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-15T02:28:28.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-15T02:28:28.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-15T02:28:28.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-15T02:28:28.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-15T02:28:28.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-15T02:28:28.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-15T02:28:28.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-09-15T02:28:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:28.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:28.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-15T02:28:28.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-15T02:28:29.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-15T02:28:29.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-15T02:28:29.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-15T02:28:29.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-15T02:28:29.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-15T02:28:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-15T02:28:29.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-15T02:28:29.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-15T02:28:29.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-15T02:28:29.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-15T02:28:29.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-15T02:28:29.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.335 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-09-15T02:28:29.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-15T02:28:29.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-15T02:28:29.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-15T02:28:29.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-15T02:28:29.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-15T02:28:29.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-15T02:28:29.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-15T02:28:29.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-15T02:28:29.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-15T02:28:29.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-15T02:28:29.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-15T02:28:29.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-15T02:28:29.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-15T02:28:29.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-15T02:28:29.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-15T02:28:29.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-15T02:28:29.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-15T02:28:29.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-15T02:28:29.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-15T02:28:29.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-15T02:28:29.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-15T02:28:29.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-15T02:28:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-15T02:28:29.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-09-15T02:28:29.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:29.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:29.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-15T02:28:29.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-15T02:28:30.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-15T02:28:30.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-15T02:28:30.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-15T02:28:30.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-15T02:28:30.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-15T02:28:30.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-15T02:28:30.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-15T02:28:30.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-15T02:28:30.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-15T02:28:30.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-15T02:28:30.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-15T02:28:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-15T02:28:30.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-15T02:28:30.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-15T02:28:30.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-15T02:28:30.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-15T02:28:30.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-15T02:28:30.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-15T02:28:30.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-15T02:28:30.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-15T02:28:30.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-15T02:28:30.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-15T02:28:30.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-15T02:28:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-15T02:28:30.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:28:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-15T02:28:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-09-15T02:28:30.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:28:30.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-15T02:28:30.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:28:30.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:28:31.160 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:28:31.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T02:28:31.835 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T02:28:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T02:28:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:28:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:28:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:28:31.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:28:31.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:28:31.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:28:31.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:28:31.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:28:31.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:28:31.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:28:31.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:28:31.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:28:31.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:28:31.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:28:31.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:28:31.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:28:31.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:28:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:28:31.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:28:31.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:28:31.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:28:31.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:28:32.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:28:32.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:32.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:32.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:28:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:28:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:28:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:28:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:28:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:28:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:28:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:28:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:28:32.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:28:32.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:28:32.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:28:32.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:28:32.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:28:32.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:28:32.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:28:32.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:28:32.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:28:32.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:28:32.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:28:32.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:28:32.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:28:32.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:28:32.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:32.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:32.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:28:32.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:28:32.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:28:32.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:28:32.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:28:32.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:32.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:32.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:28:32.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:28:32.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:28:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:28:32.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:28:32.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:28:32.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:28:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:28:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:32.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:28:32.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:28:32.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:28:32.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:28:32.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:28:32.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:28:32.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:28:32.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:28:32.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:28:32.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:28:32.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:28:32.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:28:32.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:28:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:28:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:28:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:28:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:28:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:28:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:28:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:28:33.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:28:33.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:28:33.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:28:33.589 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:28:33.589 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:33.589 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:33.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:28:33.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:28:33.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:28:33.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:28:33.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:28:33.906 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:28:33.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:28:34.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:28:34.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:28:34.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:28:34.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:28:34.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:34.918 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:34.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:28:34.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:28:34.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:28:34.985 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:28:34.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:28:34.994 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:28:32.404+0000 7faa60ca4900 0 load: jerasure load: lrc 2024-09-15T02:28:34.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-09-15T02:28:34.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-09-15T02:28:34.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2024-09-15T02:28:34.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2024-09-15T02:28:34.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T02:28:34.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:28:34.996 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T02:28:34.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T02:28:35.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:28:35.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:28:35.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:28:35.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:28:35.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:28:35.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:28:35.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:28:35.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d2dedd0-3c5d-4b36-b07e-f387a61aee58 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1d2dedd0-3c5d-4b36-b07e-f387a61aee58' 2024-09-15T02:28:35.003 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 1d2dedd0-3c5d-4b36-b07e-f387a61aee58 2024-09-15T02:28:35.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:28:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTRuZmLS7bABAAT29KuAft5WeNnoZt7nFXVg== 2024-09-15T02:28:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTRuZmLS7bABAAT29KuAft5WeNnoZt7nFXVg=="}' 2024-09-15T02:28:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d2dedd0-3c5d-4b36-b07e-f387a61aee58 -i td/test-erasure-eio/0/new.json 2024-09-15T02:28:35.284 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:28:35.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:28:35.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTRuZmLS7bABAAT29KuAft5WeNnoZt7nFXVg== --osd-uuid 1d2dedd0-3c5d-4b36-b07e-f387a61aee58 2024-09-15T02:28:35.320 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:35.318+0000 7f1956011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:35.331 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:35.329+0000 7f1956011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:35.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:35.331+0000 7f1956011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:35.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:35.331+0000 7f1956011440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:28:35.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:35.331+0000 7f1956011440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:28:35.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:35.331+0000 7f1956011440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:28:35.370 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:35.369+0000 7f1956011440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:28:41.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:28:41.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:28:41.352 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:28:41.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:28:41.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:28:41.855 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:28:41.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:28:41.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:28:41.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:28:41.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:28:41.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:28:41.879 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:41.878+0000 7fec8c30f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:41.881 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:41.880+0000 7fec8c30f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:41.882 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:41.881+0000 7fec8c30f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:28:42.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:28:42.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:28:42.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:28:42.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:28:42.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:28:42.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:42.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:28:42.250 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:28:42.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:42.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:42.404 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:42.403+0000 7fec8c30f440 -1 Falling back to public interface 2024-09-15T02:28:42.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:43.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:43.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:43.642 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:28:43.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:28:43.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:43.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:44.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:44.922 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:28:44.920+0000 7fec8c30f440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:28:45.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:45.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:45.002 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:28:45.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:28:45.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:45.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:45.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:46.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:46.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:46.528 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:28:46.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:28:46.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:46.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:46.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:47.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:47.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:47.888 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:28:47.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:28:47.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:47.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:48.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:49.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:49.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:49.252 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:28:49.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:28:49.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:49.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:49.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:50.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:50.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:50.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:28:50.673 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:28:50.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:50.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:51.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:52.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:52.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:52.050 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:28:52.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:28:52.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:52.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:52.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:53.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:53.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:53.411 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:28:53.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:28:53.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:53.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:53.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:54.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:54.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:54.814 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:28:54.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:28:54.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:54.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:55.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:56.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:56.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:56.192 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:28:56.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:28:56.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:56.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:56.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:57.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:57.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:57.594 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:28:57.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:28:57.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:57.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:28:58.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:28:59.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:28:59.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:28:59.348 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:28:59.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:28:59.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:28:59.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:29:00.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:01.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:01.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:01.252 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:29:01.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:29:01.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:01.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:29:01.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:02.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:02.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:02.634 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:29:02.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:29:02.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:02.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:29:03.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:04.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:04.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:04.273 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:29:04.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:29:04.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:04.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:29:05.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:06.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:06.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:06.109 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:29:06.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:29:06.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:06.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:29:07.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:08.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:08.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:08.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:29:08.005 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:29:08.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:08.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:29:08.457 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/113382,v1:127.0.0.1:6803/113382] [v2:127.0.0.1:6804/113382,v1:127.0.0.1:6805/113382] exists,up 1d2dedd0-3c5d-4b36-b07e-f387a61aee58 2024-09-15T02:29:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:29:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:29:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:29:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:29:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:29:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:29:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:29:08.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:29:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:29:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:29:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:29:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:29:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:29:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:29:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:29:08.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:29:08.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=22943ef0-df87-473f-b538-14fb519d4bc3 2024-09-15T02:29:08.465 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 22943ef0-df87-473f-b538-14fb519d4bc3 2024-09-15T02:29:08.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 22943ef0-df87-473f-b538-14fb519d4bc3' 2024-09-15T02:29:08.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:29:08.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0RuZm+wBUHBAA1RJIuQquHoz4g5/hN8YaVA== 2024-09-15T02:29:08.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0RuZm+wBUHBAA1RJIuQquHoz4g5/hN8YaVA=="}' 2024-09-15T02:29:08.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 22943ef0-df87-473f-b538-14fb519d4bc3 -i td/test-erasure-eio/1/new.json 2024-09-15T02:29:08.901 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:29:08.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:29:08.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0RuZm+wBUHBAA1RJIuQquHoz4g5/hN8YaVA== --osd-uuid 22943ef0-df87-473f-b538-14fb519d4bc3 2024-09-15T02:29:08.938 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:08.937+0000 7f956fc89440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:08.939 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:08.939+0000 7f956fc89440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:08.940 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:08.939+0000 7f956fc89440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:08.941 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:08.940+0000 7f956fc89440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:29:08.941 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:08.940+0000 7f956fc89440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:29:08.941 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:08.940+0000 7f956fc89440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:29:09.016 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:09.015+0000 7f956fc89440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:29:14.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:29:14.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:29:14.872 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:29:14.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:29:14.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:29:15.380 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:29:15.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:29:15.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:29:15.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:29:15.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:29:15.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:29:15.403 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:15.402+0000 7ffb4681d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:15.405 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:15.404+0000 7ffb4681d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:15.406 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:15.405+0000 7ffb4681d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:15.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:29:15.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:29:15.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:29:15.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:29:15.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:29:15.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:15.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:29:15.753 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:29:15.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:15.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:15.924 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:15.923+0000 7ffb4681d440 -1 Falling back to public interface 2024-09-15T02:29:16.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:17.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:17.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:17.114 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:29:17.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:29:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:17.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:18.443 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:18.442+0000 7ffb4681d440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:29:18.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:18.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:18.484 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:29:18.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:29:18.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:18.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:19.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:20.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:20.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:20.098 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:29:20.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:29:20.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:20.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:20.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:21.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:21.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:21.500 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:29:21.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:29:21.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:21.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:21.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:22.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:22.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:22.962 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:29:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:29:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:23.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:24.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:24.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:24.421 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:29:24.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:29:24.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:24.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:24.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:25.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:25.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:25.781 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:29:25.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:29:25.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:25.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:26.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:27.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:27.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:27.314 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:29:27.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:29:27.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:27.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:27.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:28.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:28.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:28.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:29:28.714 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:29:28.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:28.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:29.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:30.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:30.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:30.139 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:29:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:29:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:30.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:31.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:31.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:31.702 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:29:31.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:29:31.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:31.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:32.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:33.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:33.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:33.933 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:29:33.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:29:33.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:33.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:35.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:35.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:35.701 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:29:35.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:29:35.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:35.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:36.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:37.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:37.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:37.612 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:29:37.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:29:37.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:37.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:39.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:39.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:39.465 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:29:39.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:29:39.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:39.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:40.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:41.272 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:29:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:29:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:41.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:42.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:42.708 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:29:42.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:42.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:29:42.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:42.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:29:43.063 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/115137,v1:127.0.0.1:6811/115137] [v2:127.0.0.1:6812/115137,v1:127.0.0.1:6813/115137] exists,up 22943ef0-df87-473f-b538-14fb519d4bc3 2024-09-15T02:29:43.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:29:43.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:29:43.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:29:43.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:29:43.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:29:43.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:29:43.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:29:43.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:29:43.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:29:43.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:29:43.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:29:43.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cad8e3f3-d215-47ec-a8f5-f4a5f552643b 2024-09-15T02:29:43.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cad8e3f3-d215-47ec-a8f5-f4a5f552643b' 2024-09-15T02:29:43.071 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 cad8e3f3-d215-47ec-a8f5-f4a5f552643b 2024-09-15T02:29:43.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:29:43.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXRuZmgcrFBBAA4V0qGvJK9EuOZb42yxXi3w== 2024-09-15T02:29:43.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXRuZmgcrFBBAA4V0qGvJK9EuOZb42yxXi3w=="}' 2024-09-15T02:29:43.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cad8e3f3-d215-47ec-a8f5-f4a5f552643b -i td/test-erasure-eio/2/new.json 2024-09-15T02:29:43.552 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:29:43.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:29:43.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXRuZmgcrFBBAA4V0qGvJK9EuOZb42yxXi3w== --osd-uuid cad8e3f3-d215-47ec-a8f5-f4a5f552643b 2024-09-15T02:29:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:43.587+0000 7f9f5c426440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:43.590 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:43.589+0000 7f9f5c426440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:43.590+0000 7f9f5c426440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:43.590+0000 7f9f5c426440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:29:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:43.590+0000 7f9f5c426440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:29:43.591 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:43.590+0000 7f9f5c426440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:29:43.667 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:43.667+0000 7f9f5c426440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:29:49.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:29:49.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:29:49.540 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:29:49.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:29:49.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:29:50.020 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:29:50.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:29:50.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:29:50.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:29:50.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:29:50.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:29:50.044 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:50.042+0000 7f67e180d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:50.046 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:50.044+0000 7f67e180d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:50.047 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:50.045+0000 7f67e180d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:29:50.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:29:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:29:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:29:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:29:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:29:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:29:50.370 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:29:50.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:50.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:29:50.562 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:50.560+0000 7f67e180d440 -1 Falling back to public interface 2024-09-15T02:29:50.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:51.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:51.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:51.720 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:29:51.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:29:51.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:51.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:29:52.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:53.063 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:29:53.060+0000 7f67e180d440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:29:53.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:29:53.117 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:29:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:53.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:29:53.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:54.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:54.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:54.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:29:54.725 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:29:54.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:54.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:29:55.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:56.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:56.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:56.128 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:29:56.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:29:56.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:56.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:29:56.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:57.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:57.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:57.517 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:29:57.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:29:57.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:57.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:29:57.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:29:58.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:29:58.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:29:58.902 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:29:58.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:29:58.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:29:58.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:29:59.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:00.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:00.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:00.271 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:30:00.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:30:00.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:00.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:00.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:01.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:01.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:01.631 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:30:01.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:30:01.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:01.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:02.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:03.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:03.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:03.169 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:30:03.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:30:03.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:03.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:03.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:04.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:04.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:04.587 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:30:04.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:30:04.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:04.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:05.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:06.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:06.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:06.370 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:30:06.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:30:06.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:06.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:07.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:08.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:08.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:08.277 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:30:08.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:30:08.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:08.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:09.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:10.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:10.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:10.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:30:10.050 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:30:10.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:10.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:10.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:11.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:11.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:11.927 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:30:11.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:30:11.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:11.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:13.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:14.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:14.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:14.016 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:30:14.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:30:14.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:14.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:14.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:15.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:15.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:15.743 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:30:15.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:30:15.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:15.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:30:16.314 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/116891,v1:127.0.0.1:6819/116891] [v2:127.0.0.1:6820/116891,v1:127.0.0.1:6821/116891] exists,up cad8e3f3-d215-47ec-a8f5-f4a5f552643b 2024-09-15T02:30:16.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:30:16.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:30:16.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:30:16.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:30:16.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:30:16.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7fc4552c-26ab-4833-8138-f2df546c7528 2024-09-15T02:30:16.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7fc4552c-26ab-4833-8138-f2df546c7528' 2024-09-15T02:30:16.322 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 7fc4552c-26ab-4833-8138-f2df546c7528 2024-09-15T02:30:16.322 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:30:16.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4RuZmgs+4ExAAej0ADBVoFaYpbrXBYvtKhA== 2024-09-15T02:30:16.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4RuZmgs+4ExAAej0ADBVoFaYpbrXBYvtKhA=="}' 2024-09-15T02:30:16.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7fc4552c-26ab-4833-8138-f2df546c7528 -i td/test-erasure-eio/3/new.json 2024-09-15T02:30:16.931 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:30:16.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:30:16.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4RuZmgs+4ExAAej0ADBVoFaYpbrXBYvtKhA== --osd-uuid 7fc4552c-26ab-4833-8138-f2df546c7528 2024-09-15T02:30:16.967 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:16.966+0000 7fbe1511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:17.068 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:17.067+0000 7fbe1511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:17.069 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:17.068+0000 7fbe1511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:17.070 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:17.069+0000 7fbe1511f440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:30:17.070 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:17.069+0000 7fbe1511f440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:30:17.070 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:17.069+0000 7fbe1511f440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:30:17.178 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:17.177+0000 7fbe1511f440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:30:23.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:30:23.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:30:23.126 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:30:23.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:30:23.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:30:23.621 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:30:23.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:30:23.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:30:23.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:30:23.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:30:23.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:30:23.644 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:23.643+0000 7fa634724440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:23.658 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:23.657+0000 7fa634724440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:23.658+0000 7fa634724440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:30:23.991 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:30:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:24.184 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:24.183+0000 7fa634724440 -1 Falling back to public interface 2024-09-15T02:30:24.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:25.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:25.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:25.342 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:30:25.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:30:25.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:25.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:25.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:26.662 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:26.661+0000 7fa634724440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:30:26.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:26.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:26.698 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:30:26.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:30:26.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:26.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:27.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:28.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:28.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:28.123 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:30:28.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:30:28.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:28.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:28.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:29.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:29.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:29.570 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:30:29.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:30:29.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:29.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:30.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:31.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:31.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:31.047 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:30:31.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:30:31.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:31.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:32.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:32.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:32.511 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:30:32.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:30:32.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:32.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:32.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:33.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:33.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:33.894 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:30:33.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:30:33.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:33.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:34.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:35.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:35.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:35.253 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:30:35.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:30:35.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:35.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:35.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:36.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:36.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:36.707 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:30:36.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:30:36.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:36.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:37.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:38.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:38.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:38.067 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:30:38.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:30:38.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:38.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:38.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:39.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:39.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:39.586 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:30:39.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:30:39.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:39.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:40.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:41.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:41.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:41.811 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:30:41.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:30:41.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:41.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:42.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:43.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:43.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:43.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:30:43.309 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:30:43.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:43.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:43.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:44.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:44.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:44.760 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:30:44.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:30:44.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:44.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:45.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:46.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:46.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:46.517 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:30:46.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:30:46.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:46.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:47.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:48.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:48.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:48.330 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:30:48.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:30:48.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:48.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:49.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:30:50.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:30:50.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:50.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:30:50.115 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:30:50.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:50.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:30:50.510 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/118620,v1:127.0.0.1:6827/118620] [v2:127.0.0.1:6828/118620,v1:127.0.0.1:6829/118620] exists,up 7fc4552c-26ab-4833-8138-f2df546c7528 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T02:30:50.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:30:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:30:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:30:50.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:30:50.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T02:30:50.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:30:50.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90c8a2c3-6de6-40c8-842c-a4d60f3cd481 2024-09-15T02:30:50.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 90c8a2c3-6de6-40c8-842c-a4d60f3cd481' 2024-09-15T02:30:50.517 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 90c8a2c3-6de6-40c8-842c-a4d60f3cd481 2024-09-15T02:30:50.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:30:50.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaRuZmX9luHxAAOixIUE7Bk0TW3hFiW9wE7Q== 2024-09-15T02:30:50.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaRuZmX9luHxAAOixIUE7Bk0TW3hFiW9wE7Q=="}' 2024-09-15T02:30:50.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90c8a2c3-6de6-40c8-842c-a4d60f3cd481 -i td/test-erasure-eio/4/new.json 2024-09-15T02:30:51.068 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:30:51.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T02:30:51.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaRuZmX9luHxAAOixIUE7Bk0TW3hFiW9wE7Q== --osd-uuid 90c8a2c3-6de6-40c8-842c-a4d60f3cd481 2024-09-15T02:30:51.104 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:51.103+0000 7f6a86121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:51.246 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:51.244+0000 7f6a86121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:51.249 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:51.246+0000 7f6a86121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:51.249 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:51.246+0000 7f6a86121440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:30:51.249 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:51.246+0000 7f6a86121440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:30:51.249 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:51.246+0000 7f6a86121440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:30:51.320 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:51.319+0000 7f6a86121440 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T02:30:58.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T02:30:58.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:30:58.488 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:30:58.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:30:58.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:30:58.976 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:30:58.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:30:58.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:30:58.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:30:58.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:30:58.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:30:58.999 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:58.997+0000 7fbf61626440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:59.000 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:59.000+0000 7fbf61626440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:59.002 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:59.001+0000 7fbf61626440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:30:59.351 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:30:59.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:30:59.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:30:59.516 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:30:59.515+0000 7fbf61626440 -1 Falling back to public interface 2024-09-15T02:30:59.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:00.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:00.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:00.722 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:31:00.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:31:00.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:00.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:01.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:02.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:02.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:02.081 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:31:02.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:31:02.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:02.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:02.086 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:02.084+0000 7fbf61626440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:31:02.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:03.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:03.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:03.579 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:31:03.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:31:03.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:03.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:04.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:05.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:05.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:05.148 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:31:05.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:31:05.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:05.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:05.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:06.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:06.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:06.522 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:31:06.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:31:06.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:06.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:06.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:07.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:07.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:07.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:31:07.993 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:31:07.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:07.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:08.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:09.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:09.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:09.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:31:09.459 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:31:09.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:09.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:09.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:10.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:10.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:10.895 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:31:10.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:31:10.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:10.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:11.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:12.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:12.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:12.794 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:31:12.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:31:12.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:12.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:12.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:13.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:13.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:13.656 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:31:13.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:31:13.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:13.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:14.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:15.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:15.018 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:31:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:31:15.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:15.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:15.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:16.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:16.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:16.411 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:31:16.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:31:16.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:16.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:17.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:18.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:18.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:18.067 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:31:18.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:31:18.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:18.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:18.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:19.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:19.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:19.976 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:31:19.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:31:19.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:19.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:20.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:21.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:21.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:21.933 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:31:21.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:31:21.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:21.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:22.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:23.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:23.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:23.636 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:31:23.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:31:23.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:23.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:24.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:25.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:25.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:25.135 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:31:25.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:31:25.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:25.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:25.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:26.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:26.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:26.697 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:31:26.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:31:26.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:26.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:31:27.044 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/120390,v1:127.0.0.1:6835/120390] [v2:127.0.0.1:6836/120390,v1:127.0.0.1:6837/120390] exists,up 90c8a2c3-6de6-40c8-842c-a4d60f3cd481 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:31:27.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:31:27.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T02:31:27.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:31:27.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef7a5cb3-dc2c-4fd6-ac99-5e09aecd074a 2024-09-15T02:31:27.051 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 ef7a5cb3-dc2c-4fd6-ac99-5e09aecd074a 2024-09-15T02:31:27.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ef7a5cb3-dc2c-4fd6-ac99-5e09aecd074a' 2024-09-15T02:31:27.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:31:27.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/RuZmGDmkAxAAyBNUwr6a1NZw5iBISdqYVQ== 2024-09-15T02:31:27.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/RuZmGDmkAxAAyBNUwr6a1NZw5iBISdqYVQ=="}' 2024-09-15T02:31:27.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef7a5cb3-dc2c-4fd6-ac99-5e09aecd074a -i td/test-erasure-eio/5/new.json 2024-09-15T02:31:27.846 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:31:27.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T02:31:27.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/RuZmGDmkAxAAyBNUwr6a1NZw5iBISdqYVQ== --osd-uuid ef7a5cb3-dc2c-4fd6-ac99-5e09aecd074a 2024-09-15T02:31:27.883 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:27.882+0000 7f85fc009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:31:27.884 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:27.883+0000 7f85fc009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:31:27.885 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:27.884+0000 7f85fc009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:31:27.886 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:27.885+0000 7f85fc009440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:31:27.886 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:27.885+0000 7f85fc009440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:31:27.886 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:27.885+0000 7f85fc009440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:31:27.980 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:27.979+0000 7f85fc009440 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T02:31:34.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T02:31:34.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:31:34.854 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-15T02:31:34.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T02:31:34.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:31:35.367 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-15T02:31:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T02:31:35.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:31:35.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:31:35.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:31:35.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:31:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:35.388+0000 7f9d4960f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:31:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:35.390+0000 7f9d4960f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:31:35.392 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:35.391+0000 7f9d4960f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:31:35.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T02:31:35.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:31:35.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T02:31:35.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:31:35.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:31:35.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:35.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:31:35.716 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:31:35.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:35.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:35.914 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:35.913+0000 7f9d4960f440 -1 Falling back to public interface 2024-09-15T02:31:36.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:37.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:37.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:37.079 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:31:37.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:31:37.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:37.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:37.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:38.411 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:31:38.409+0000 7f9d4960f440 -1 osd.5 0 log_to_monitors true 2024-09-15T02:31:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:38.464 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:31:38.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:31:38.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:38.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:38.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:39.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:39.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:39.918 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:31:39.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:31:39.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:39.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:40.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:41.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:41.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:41.429 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:31:41.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:31:41.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:41.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:41.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:42.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:42.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:42.877 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:31:42.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:31:42.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:42.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:43.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:44.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:44.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:44.297 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:31:44.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:31:44.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:44.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:44.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:45.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:45.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:45.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:31:45.662 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:31:45.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:45.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:46.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:47.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:47.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:47.092 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:31:47.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:31:47.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:47.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:47.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:48.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:48.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:48.557 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:31:48.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:31:48.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:48.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:48.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:49.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:49.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:49.940 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:31:49.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:31:49.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:49.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:50.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:51.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:51.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:51.354 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:31:51.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:31:51.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:51.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:52.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:53.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:53.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:53.240 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:31:53.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:31:53.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:53.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:53.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:54.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:54.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:54.747 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:31:54.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:31:54.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:54.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:55.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:56.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:56.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:56.413 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:31:56.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:31:56.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:56.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:57.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:31:58.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:31:58.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:31:58.182 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:31:58.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:31:58.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:31:58.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:31:59.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:00.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:00.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:00.094 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:32:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:32:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:00.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:32:00.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:01.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:01.871 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:32:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:32:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:01.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:32:02.351 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/122169,v1:127.0.0.1:6843/122169] [v2:127.0.0.1:6844/122169,v1:127.0.0.1:6845/122169] exists,up ef7a5cb3-dc2c-4fd6-ac99-5e09aecd074a 2024-09-15T02:32:02.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:32:02.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:32:02.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:32:02.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:32:02.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00704911-649b-4941-916a-c4b01362852b 2024-09-15T02:32:02.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 00704911-649b-4941-916a-c4b01362852b' 2024-09-15T02:32:02.359 INFO:tasks.workunit.client.0.smithi130.stdout:add osd6 00704911-649b-4941-916a-c4b01362852b 2024-09-15T02:32:02.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:32:02.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiR+ZmR4L6FRAAwvLj19G07RFfdfVl31XZ5g== 2024-09-15T02:32:02.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiR+ZmR4L6FRAAwvLj19G07RFfdfVl31XZ5g=="}' 2024-09-15T02:32:02.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00704911-649b-4941-916a-c4b01362852b -i td/test-erasure-eio/6/new.json 2024-09-15T02:32:03.121 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:32:03.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T02:32:03.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiR+ZmR4L6FRAAwvLj19G07RFfdfVl31XZ5g== --osd-uuid 00704911-649b-4941-916a-c4b01362852b 2024-09-15T02:32:03.157 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:03.156+0000 7f7e4b839440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:03.183 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:03.181+0000 7f7e4b839440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:03.183 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:03.182+0000 7f7e4b839440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:03.184 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:03.183+0000 7f7e4b839440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:32:03.184 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:03.183+0000 7f7e4b839440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:32:03.184 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:03.183+0000 7f7e4b839440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:32:03.275 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:03.274+0000 7f7e4b839440 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T02:32:09.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T02:32:09.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:32:09.601 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd6 key to auth repository 2024-09-15T02:32:09.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-15T02:32:09.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:32:10.146 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.6 2024-09-15T02:32:10.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-15T02:32:10.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:32:10.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:32:10.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:32:10.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:32:10.168 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:10.167+0000 7f7678325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:10.170 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:10.169+0000 7f7678325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:10.171 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:10.170+0000 7f7678325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:10.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-15T02:32:10.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:32:10.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-15T02:32:10.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:32:10.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:32:10.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:10.496 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:32:10.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:32:10.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:10.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:10.684 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:10.683+0000 7f7678325440 -1 Falling back to public interface 2024-09-15T02:32:10.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:11.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:11.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:11.859 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:32:11.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:32:11.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:11.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:12.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:13.182 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:13.181+0000 7f7678325440 -1 osd.6 0 log_to_monitors true 2024-09-15T02:32:13.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:13.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:13.215 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:32:13.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:32:13.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:13.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:13.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:14.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:14.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:14.624 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:32:14.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:32:14.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:14.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:15.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:16.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:16.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:16.117 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:32:16.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:32:16.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:16.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:16.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:17.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:17.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:17.783 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:32:17.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:32:17.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:17.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:18.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:19.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:19.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:19.240 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:32:19.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:32:19.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:19.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:19.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:20.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:20.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:20.638 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:32:20.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:32:20.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:20.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:21.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:22.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:22.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:22.109 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:32:22.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:32:22.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:22.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:22.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:23.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:23.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:23.636 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:32:23.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:32:23.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:23.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:24.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:25.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:25.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:25.124 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:32:25.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:32:25.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:25.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:25.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:26.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:26.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:26.565 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:32:26.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:32:26.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:26.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:26.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:27.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:27.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:27.953 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:32:27.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:32:27.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:27.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:28.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:29.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:29.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:29.739 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:32:29.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:32:29.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:29.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:31.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:32.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:32.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:32.012 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:32:32.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:32:32.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:32.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:32.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:33.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:33.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:33.823 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:32:33.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:32:33.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:33.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:34.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:35.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:35.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:35.626 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:32:35.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:32:35.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:35.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:36.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:37.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:37.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:37.492 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:32:37.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:32:37.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:37.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:32:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/123923,v1:127.0.0.1:6851/123923] [v2:127.0.0.1:6852/123923,v1:127.0.0.1:6853/123923] exists,up 00704911-649b-4941-916a-c4b01362852b 2024-09-15T02:32:37.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:32:37.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:32:37.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:32:37.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:32:37.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:32:37.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:32:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:32:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:32:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:32:37.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:32:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:32:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:32:37.996 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:32:38.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:32:38.008 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:28:42.407+0000 7fec8c30f440 0 load: jerasure load: lrc 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:32:38.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T02:32:39.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:32:39.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:32:40.001 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:32:40.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:32:41.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:32:41.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:32:41.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:32:41.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:32:41.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:32:41.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:32:41.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:32:41.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:32:41.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:32:41.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:32:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:32:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:32:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:32:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:32:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:32:41.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:41.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:32:41.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803825 2024-09-15T02:32:41.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803825 2024-09-15T02:32:41.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825' 2024-09-15T02:32:41.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:41.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:32:41.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574886 2024-09-15T02:32:41.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574886 2024-09-15T02:32:41.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574886' 2024-09-15T02:32:41.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:41.861 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:32:42.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378658 2024-09-15T02:32:42.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378658 2024-09-15T02:32:42.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574886 2-81604378658' 2024-09-15T02:32:42.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:42.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:32:42.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084315 2024-09-15T02:32:42.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084315 2024-09-15T02:32:42.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574886 2-81604378658 3-120259084315' 2024-09-15T02:32:42.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:42.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:32:42.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691863 2024-09-15T02:32:42.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691863 2024-09-15T02:32:42.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574886 2-81604378658 3-120259084315 4-171798691863' 2024-09-15T02:32:42.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:42.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:32:42.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364815 2024-09-15T02:32:42.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364815 2024-09-15T02:32:42.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574886 2-81604378658 3-120259084315 4-171798691863 5-214748364815' 2024-09-15T02:32:42.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:42.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:32:42.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037771 2024-09-15T02:32:42.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037771 2024-09-15T02:32:42.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574886 2-81604378658 3-120259084315 4-171798691863 5-214748364815 6-257698037771' 2024-09-15T02:32:42.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:42.535 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803825 2024-09-15T02:32:42.535 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:42.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:32:42.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803825 2024-09-15T02:32:42.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:42.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803825 2024-09-15T02:32:42.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-15T02:32:42.538 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803825 2024-09-15T02:32:42.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:32:42.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803823 -lt 25769803825 2024-09-15T02:32:42.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:32:43.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:32:43.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:32:44.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803825 -lt 25769803825 2024-09-15T02:32:44.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:44.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574886 2024-09-15T02:32:44.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:44.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:32:44.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574886 2024-09-15T02:32:44.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:44.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574886 2024-09-15T02:32:44.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574886' 2024-09-15T02:32:44.246 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574886 2024-09-15T02:32:44.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:32:44.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574887 -lt 55834574886 2024-09-15T02:32:44.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:44.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378658 2024-09-15T02:32:44.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:44.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:32:44.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378658 2024-09-15T02:32:44.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:44.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378658 2024-09-15T02:32:44.597 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378658 2024-09-15T02:32:44.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378658' 2024-09-15T02:32:44.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:32:44.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378658 -lt 81604378658 2024-09-15T02:32:44.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:44.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084315 2024-09-15T02:32:44.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:44.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:32:44.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084315 2024-09-15T02:32:44.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:44.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084315 2024-09-15T02:32:44.962 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084315 2024-09-15T02:32:44.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084315' 2024-09-15T02:32:44.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:32:45.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084315 -lt 120259084315 2024-09-15T02:32:45.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:45.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691863 2024-09-15T02:32:45.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:45.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:32:45.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691863 2024-09-15T02:32:45.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:45.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691863 2024-09-15T02:32:45.319 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 171798691863 2024-09-15T02:32:45.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691863' 2024-09-15T02:32:45.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:32:45.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691863 -lt 171798691863 2024-09-15T02:32:45.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:45.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-214748364815 2024-09-15T02:32:45.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:45.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:32:45.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-214748364815 2024-09-15T02:32:45.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:45.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364815 2024-09-15T02:32:45.698 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 214748364815 2024-09-15T02:32:45.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 214748364815' 2024-09-15T02:32:45.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:32:46.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364816 -lt 214748364815 2024-09-15T02:32:46.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:46.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-257698037771 2024-09-15T02:32:46.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:46.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:32:46.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-257698037771 2024-09-15T02:32:46.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:46.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037771 2024-09-15T02:32:46.091 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 257698037771 2024-09-15T02:32:46.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 257698037771' 2024-09-15T02:32:46.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:32:46.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037772 -lt 257698037771 2024-09-15T02:32:46.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:32:46.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:32:46.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:32:46.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:32:46.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:32:46.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:32:46.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:32:46.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:32:46.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:32:46.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:32:46.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:32:47.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:32:47.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:32:47.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:32:47.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:32:47.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T02:32:48.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2024-09-15T02:32:48.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T02:32:48.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:32:48.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:32:48.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:32:48.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:32:48.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:32:48.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:32:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:32:48.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T02:32:49.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-09-15T02:32:49.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:32:49.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:32:49.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=mdata 2024-09-15T02:32:49.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:32:49.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:32:49.503 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:32:49.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2024-09-15T02:32:49.942 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:32:49.942 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:32:49.942 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:32:49.942 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:32:49.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2024-09-15T02:32:49.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-09-15T02:32:49.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:32:49.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=5 2024-09-15T02:32:49.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:32:49.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/5/type 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-15T02:32:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:32:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:32:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:32:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.5.asok 2024-09-15T02:32:49.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:32:50.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:32:50.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:32:50.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:32:50.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.5 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.5.asok 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:32:50.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-09-15T02:32:50.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:32:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:32:50.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:32:50.110 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:32:50.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T02:32:50.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:32:50.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T02:32:50.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T02:32:50.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:32:50.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:32:50.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:32:50.130 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:50.129+0000 7ff06300d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:50.132 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:50.131+0000 7ff06300d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:50.133 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:50.132+0000 7ff06300d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:32:50.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T02:32:50.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:32:50.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:32:50.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:32:50.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:32:50.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:50.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:32:50.458 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:32:50.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:50.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:32:50.652 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:50.650+0000 7ff06300d440 -1 Falling back to public interface 2024-09-15T02:32:50.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:51.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:51.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:51.805 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:32:51.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:32:51.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:51.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:32:52.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:53.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:53.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:53.167 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:32:53.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:32:53.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:53.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:32:53.688 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:32:53.687+0000 7ff06300d440 -1 osd.3 68 log_to_monitors true 2024-09-15T02:32:53.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:32:54.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:32:54.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:32:54.702 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:32:54.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:32:54.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:32:54.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:32:55.057 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [28,68) [v2:127.0.0.1:6826/125535,v1:127.0.0.1:6827/125535] [v2:127.0.0.1:6828/125535,v1:127.0.0.1:6829/125535] exists,up 7fc4552c-26ab-4833-8138-f2df546c7528 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:32:55.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:32:55.060 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:32:55.060 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:32:55.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:32:55.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:32:55.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:32:55.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:32:55.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:32:55.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:32:55.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:32:55.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:32:55.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:32:55.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:32:55.695 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:32:55.695 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:32:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:32:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:32:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:32:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:32:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:32:55.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:55.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:32:55.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803830 2024-09-15T02:32:55.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803830 2024-09-15T02:32:55.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803830' 2024-09-15T02:32:55.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:55.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:32:56.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574891 2024-09-15T02:32:56.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574891 2024-09-15T02:32:56.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803830 1-55834574891' 2024-09-15T02:32:56.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:56.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:32:56.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378662 2024-09-15T02:32:56.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378662 2024-09-15T02:32:56.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803830 1-55834574891 2-81604378662' 2024-09-15T02:32:56.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:56.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:32:56.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645315 2024-09-15T02:32:56.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645315 2024-09-15T02:32:56.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803830 1-55834574891 2-81604378662 3-309237645315' 2024-09-15T02:32:56.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:56.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:32:56.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691867 2024-09-15T02:32:56.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691867 2024-09-15T02:32:56.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803830 1-55834574891 2-81604378662 3-309237645315 4-171798691867' 2024-09-15T02:32:56.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:56.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:32:56.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364820 2024-09-15T02:32:56.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364820 2024-09-15T02:32:56.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803830 1-55834574891 2-81604378662 3-309237645315 4-171798691867 5-214748364820' 2024-09-15T02:32:56.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:32:56.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:32:56.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037776 2024-09-15T02:32:56.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037776 2024-09-15T02:32:56.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803830 1-55834574891 2-81604378662 3-309237645315 4-171798691867 5-214748364820 6-257698037776' 2024-09-15T02:32:56.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:56.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803830 2024-09-15T02:32:56.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:56.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:32:56.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803830 2024-09-15T02:32:56.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:56.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803830 2024-09-15T02:32:56.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-09-15T02:32:56.589 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803830 2024-09-15T02:32:56.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:32:56.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803828 -lt 25769803830 2024-09-15T02:32:56.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:32:57.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:32:57.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:32:58.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803830 -lt 25769803830 2024-09-15T02:32:58.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:58.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574891 2024-09-15T02:32:58.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:58.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:32:58.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574891 2024-09-15T02:32:58.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:58.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574891 2024-09-15T02:32:58.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574891' 2024-09-15T02:32:58.296 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574891 2024-09-15T02:32:58.297 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:32:58.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574891 -lt 55834574891 2024-09-15T02:32:58.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:58.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378662 2024-09-15T02:32:58.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:58.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:32:58.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378662 2024-09-15T02:32:58.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:58.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378662 2024-09-15T02:32:58.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-09-15T02:32:58.676 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378662 2024-09-15T02:32:58.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:32:59.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378663 -lt 81604378662 2024-09-15T02:32:59.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:59.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-309237645315 2024-09-15T02:32:59.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:59.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:32:59.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-309237645315 2024-09-15T02:32:59.069 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:59.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645315 2024-09-15T02:32:59.070 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 309237645315 2024-09-15T02:32:59.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 309237645315' 2024-09-15T02:32:59.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:32:59.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645316 -lt 309237645315 2024-09-15T02:32:59.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:59.427 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691867 2024-09-15T02:32:59.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:59.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:32:59.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691867 2024-09-15T02:32:59.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:59.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691867 2024-09-15T02:32:59.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691867' 2024-09-15T02:32:59.430 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 171798691867 2024-09-15T02:32:59.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:32:59.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691868 -lt 171798691867 2024-09-15T02:32:59.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:32:59.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-214748364820 2024-09-15T02:32:59.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:32:59.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:32:59.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-214748364820 2024-09-15T02:32:59.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:32:59.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364820 2024-09-15T02:32:59.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 214748364820' 2024-09-15T02:32:59.792 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 214748364820 2024-09-15T02:32:59.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:33:00.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364821 -lt 214748364820 2024-09-15T02:33:00.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:33:00.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-257698037776 2024-09-15T02:33:00.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:33:00.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:33:00.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-257698037776 2024-09-15T02:33:00.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:33:00.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037776 2024-09-15T02:33:00.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 257698037776' 2024-09-15T02:33:00.148 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 257698037776 2024-09-15T02:33:00.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:33:00.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037777 -lt 257698037776 2024-09-15T02:33:00.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:33:00.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:33:00.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:33:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:33:01.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:33:01.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:33:01.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:33:01.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-15T02:33:01.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:33:01.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:33:01.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-15T02:33:01.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:33:01.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:33:01.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-09-15T02:33:01.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:33:01.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:33:02.412 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:33:02.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T02:33:02.990 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T02:33:03.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T02:33:03.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:33:03.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:33:03.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:33:03.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:33:03.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:33:03.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:33:03.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:33:03.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:33:03.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:33:03.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:33:03.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:33:03.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:33:03.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:33:03.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:33:03.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:33:03.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:33:03.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:33:03.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:33:03.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:33:03.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:33:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:33:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:33:03.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:33:03.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:03.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:03.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:33:03.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:33:03.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:33:03.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:33:03.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:33:03.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:33:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:33:03.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:33:03.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:33:03.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:33:03.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:33:03.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:33:03.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:33:03.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:33:03.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:33:03.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:33:03.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:33:03.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:33:03.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:33:03.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:33:03.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:33:03.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:03.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:03.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:33:03.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:33:03.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:33:03.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:33:03.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:33:03.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:03.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:03.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:33:03.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:33:03.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:33:03.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:33:03.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:33:03.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:33:03.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:33:03.773 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:33:03.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:33:03.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:33:03.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:33:03.848 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:33:03.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:33:03.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:33:03.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:33:03.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:33:03.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:33:03.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:33:03.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:33:04.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:33:04.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:33:04.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:33:04.759 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:33:04.759 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:04.759 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:04.759 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:33:04.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:33:04.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:33:04.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:33:04.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:33:05.033 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:33:05.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:33:06.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:33:06.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:33:06.046 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:33:06.046 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:33:06.046 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:06.046 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:06.046 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:33:06.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:33:06.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:33:06.113 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:33:06.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:33:06.121 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:33:03.547+0000 7f55e3cb5900 0 load: jerasure load: lrc 2024-09-15T02:33:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-09-15T02:33:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-09-15T02:33:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-09-15T02:33:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-09-15T02:33:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-09-15T02:33:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:33:06.123 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-09-15T02:33:06.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:33:06.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:33:06.128 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:33:06.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:33:06.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:33:06.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:33:06.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8002be39-3626-459a-a84f-ec78b3a36394 2024-09-15T02:33:06.130 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 8002be39-3626-459a-a84f-ec78b3a36394 2024-09-15T02:33:06.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8002be39-3626-459a-a84f-ec78b3a36394' 2024-09-15T02:33:06.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:33:06.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiR+Zmlp1jCBAAqJVsJeYM7QBOBCMc2QgzuA== 2024-09-15T02:33:06.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiR+Zmlp1jCBAAqJVsJeYM7QBOBCMc2QgzuA=="}' 2024-09-15T02:33:06.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8002be39-3626-459a-a84f-ec78b3a36394 -i td/test-erasure-eio/0/new.json 2024-09-15T02:33:06.412 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:33:06.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:33:06.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiR+Zmlp1jCBAAqJVsJeYM7QBOBCMc2QgzuA== --osd-uuid 8002be39-3626-459a-a84f-ec78b3a36394 2024-09-15T02:33:06.447 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:06.446+0000 7fdbb4a16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:06.449 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:06.449+0000 7fdbb4a16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:06.451 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:06.450+0000 7fdbb4a16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:06.451 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:06.450+0000 7fdbb4a16440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:33:06.451 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:06.450+0000 7fdbb4a16440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:33:06.451 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:06.450+0000 7fdbb4a16440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:33:06.489 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:06.488+0000 7fdbb4a16440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:33:12.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:33:12.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:33:12.531 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:33:12.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:33:12.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:33:13.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:33:13.022 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:33:13.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:33:13.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:33:13.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:33:13.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:33:13.044 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:13.043+0000 7f1020d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:13.046 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:13.045+0000 7f1020d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:13.047 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:13.046+0000 7f1020d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:13.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:33:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:33:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:33:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:33:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:33:13.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:33:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:33:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:13.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:13.572 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:13.571+0000 7f1020d21440 -1 Falling back to public interface 2024-09-15T02:33:13.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:14.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:14.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:14.720 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:33:14.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:33:14.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:14.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:15.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:16.021 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:16.020+0000 7f1020d21440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:33:16.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:16.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:16.077 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:33:16.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:33:16.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:16.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:16.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:17.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:17.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:17.574 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:33:17.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:33:17.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:17.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:17.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:18.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:18.978 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:33:18.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:18.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:33:18.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:18.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:19.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:20.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:20.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:20.390 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:33:20.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:33:20.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:20.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:20.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:21.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:21.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:21.787 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:33:21.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:33:21.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:21.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:22.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:23.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:23.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:23.158 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:33:23.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:33:23.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:23.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:23.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:24.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:24.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:24.520 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:33:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:33:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:24.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:25.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:25.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:25.960 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:33:25.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:33:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:26.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:27.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:27.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:27.348 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:33:27.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:33:27.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:27.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:28.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:29.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:29.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:29.014 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:33:29.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:33:29.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:29.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:29.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:30.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:30.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:30.694 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:33:30.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:33:30.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:30.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:31.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:32.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:32.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:32.211 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:33:32.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:33:32.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:32.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:32.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:33.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:33.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:33.688 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:33:33.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:33:33.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:33.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:34.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:35.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:35.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:35.434 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:33:35.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:33:35.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:35.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:36.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:37.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:37.054 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:33:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:33:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:37.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:37.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:38.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:38.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:38.677 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:33:38.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:33:38.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:38.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:33:39.101 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/127877,v1:127.0.0.1:6803/127877] [v2:127.0.0.1:6804/127877,v1:127.0.0.1:6805/127877] exists,up 8002be39-3626-459a-a84f-ec78b3a36394 2024-09-15T02:33:39.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:33:39.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:33:39.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:33:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:33:39.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:33:39.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d4b9bae-1711-45df-a7f6-28b8460f794c 2024-09-15T02:33:39.109 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 1d4b9bae-1711-45df-a7f6-28b8460f794c 2024-09-15T02:33:39.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1d4b9bae-1711-45df-a7f6-28b8460f794c' 2024-09-15T02:33:39.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:33:39.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDR+ZmmmYNBxAAP7caaQ/bd4MmwM8/gxM1ow== 2024-09-15T02:33:39.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDR+ZmmmYNBxAAP7caaQ/bd4MmwM8/gxM1ow=="}' 2024-09-15T02:33:39.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d4b9bae-1711-45df-a7f6-28b8460f794c -i td/test-erasure-eio/1/new.json 2024-09-15T02:33:39.653 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:33:39.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:33:39.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDR+ZmmmYNBxAAP7caaQ/bd4MmwM8/gxM1ow== --osd-uuid 1d4b9bae-1711-45df-a7f6-28b8460f794c 2024-09-15T02:33:39.690 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:39.689+0000 7f6af4009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:39.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:39.691+0000 7f6af4009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:39.692+0000 7f6af4009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:39.692+0000 7f6af4009440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:33:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:39.692+0000 7f6af4009440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:33:39.694 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:39.692+0000 7f6af4009440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:33:39.743 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:39.742+0000 7f6af4009440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:33:45.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:33:45.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:33:45.326 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:33:45.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:33:45.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:33:45.809 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:33:45.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:33:45.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:33:45.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:33:45.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:33:45.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:33:45.832 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:45.830+0000 7f1e46724440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:45.833 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:45.832+0000 7f1e46724440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:45.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:45.833+0000 7f1e46724440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:33:46.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:33:46.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:33:46.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:33:46.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:33:46.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:33:46.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:46.203 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:33:46.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:33:46.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:46.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:46.356 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:46.355+0000 7f1e46724440 -1 Falling back to public interface 2024-09-15T02:33:46.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:47.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:47.566 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:33:47.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:47.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:33:47.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:47.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:47.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:48.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:48.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:48.912 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:33:48.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:33:48.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:48.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:48.928 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:33:48.927+0000 7f1e46724440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:33:49.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:50.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:50.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:50.294 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:33:50.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:33:50.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:50.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:50.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:51.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:51.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:51.796 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:33:51.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:33:51.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:51.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:52.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:53.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:53.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:53.195 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:33:53.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:33:53.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:53.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:53.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:54.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:54.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:54.717 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:33:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:33:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:55.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:56.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:56.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:56.125 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:33:56.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:33:56.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:56.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:56.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:57.586 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:33:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:33:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:58.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:33:59.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:33:59.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:33:59.058 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:33:59.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:33:59.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:33:59.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:33:59.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:00.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:00.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:00.497 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:34:00.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:34:00.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:00.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:00.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:01.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:01.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:01.861 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:34:01.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:34:01.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:01.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:02.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:03.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:03.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:03.654 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:34:03.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:34:03.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:03.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:04.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:05.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:05.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:05.269 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:34:05.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:34:05.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:05.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:05.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:06.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:06.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:06.950 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:34:06.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:34:06.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:06.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:08.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:09.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:09.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:09.204 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:34:09.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:34:09.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:09.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:09.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:10.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:10.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:10.943 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:34:10.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:34:10.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:10.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:11.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:12.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:12.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:12.726 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:34:12.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:34:12.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:12.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:34:13.272 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/129631,v1:127.0.0.1:6811/129631] [v2:127.0.0.1:6812/129631,v1:127.0.0.1:6813/129631] exists,up 1d4b9bae-1711-45df-a7f6-28b8460f794c 2024-09-15T02:34:13.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:34:13.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:34:13.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:34:13.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:34:13.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:34:13.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:34:13.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:34:13.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:34:13.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:34:13.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:34:13.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:34:13.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e286c2b1-b575-4efd-a883-efe6728fc346 2024-09-15T02:34:13.281 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 e286c2b1-b575-4efd-a883-efe6728fc346 2024-09-15T02:34:13.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e286c2b1-b575-4efd-a883-efe6728fc346' 2024-09-15T02:34:13.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:34:13.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQClR+ZmK4c1ERAApYOuxg3qTn5Zu7d61dDTDQ== 2024-09-15T02:34:13.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQClR+ZmK4c1ERAApYOuxg3qTn5Zu7d61dDTDQ=="}' 2024-09-15T02:34:13.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e286c2b1-b575-4efd-a883-efe6728fc346 -i td/test-erasure-eio/2/new.json 2024-09-15T02:34:13.789 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:34:13.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:34:13.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQClR+ZmK4c1ERAApYOuxg3qTn5Zu7d61dDTDQ== --osd-uuid e286c2b1-b575-4efd-a883-efe6728fc346 2024-09-15T02:34:13.825 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:13.824+0000 7f641c25a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:13.826 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:13.825+0000 7f641c25a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:13.827 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:13.826+0000 7f641c25a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:13.828 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:13.827+0000 7f641c25a440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:34:13.828 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:13.827+0000 7f641c25a440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:34:13.828 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:13.827+0000 7f641c25a440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:34:13.872 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:13.871+0000 7f641c25a440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:34:19.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:34:19.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:34:19.637 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:34:19.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:34:19.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:34:20.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:34:20.121 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:34:20.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:34:20.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:34:20.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:34:20.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:34:20.144 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:20.142+0000 7f3dd7820440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:20.145 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:20.144+0000 7f3dd7820440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:20.145+0000 7f3dd7820440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:20.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:34:20.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:34:20.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:34:20.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:34:20.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:34:20.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:20.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:34:20.470 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:34:20.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:20.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:20.671 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:20.670+0000 7f3dd7820440 -1 Falling back to public interface 2024-09-15T02:34:20.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:21.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:21.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:21.827 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:34:21.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:34:21.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:21.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:23.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:23.191 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:34:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:34:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:23.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:23.382 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:23.380+0000 7f3dd7820440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:34:23.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:24.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:24.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:24.580 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:34:24.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:34:24.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:24.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:24.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:25.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:25.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:25.952 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:34:25.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:34:25.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:25.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:26.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:27.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:27.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:27.310 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:34:27.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:34:27.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:27.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:27.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:28.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:28.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:28.715 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:34:28.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:34:28.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:28.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:29.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:30.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:30.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:30.126 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:34:30.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:34:30.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:30.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:30.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:31.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:31.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:31.594 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:34:31.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:34:31.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:31.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:31.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:32.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:32.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:32.990 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:34:32.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:34:32.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:32.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:33.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:34.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:34.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:34.475 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:34:34.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:34:34.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:34.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:34.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:35.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:35.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:35.838 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:34:35.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:34:35.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:35.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:36.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:37.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:37.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:37.248 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:34:37.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:34:37.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:37.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:38.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:39.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:39.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:39.083 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:34:39.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:34:39.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:39.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:39.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:40.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:40.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:40.722 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:34:40.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:34:40.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:40.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:41.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:42.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:42.831 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:34:42.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:42.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:34:42.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:42.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:43.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:44.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:44.637 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:34:44.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:44.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:34:44.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:44.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:45.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:46.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:46.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:46.553 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:34:46.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:34:46.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:46.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:34:47.096 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/131386,v1:127.0.0.1:6819/131386] [v2:127.0.0.1:6820/131386,v1:127.0.0.1:6821/131386] exists,up e286c2b1-b575-4efd-a883-efe6728fc346 2024-09-15T02:34:47.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:34:47.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:34:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:34:47.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0dae3d3c-a54a-4e42-8a56-3c69ceb9286a 2024-09-15T02:34:47.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0dae3d3c-a54a-4e42-8a56-3c69ceb9286a' 2024-09-15T02:34:47.104 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 0dae3d3c-a54a-4e42-8a56-3c69ceb9286a 2024-09-15T02:34:47.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:34:47.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDHR+ZmAQvUBhAA4muikTvXtIA9nY2WgDrqmQ== 2024-09-15T02:34:47.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDHR+ZmAQvUBhAA4muikTvXtIA9nY2WgDrqmQ=="}' 2024-09-15T02:34:47.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0dae3d3c-a54a-4e42-8a56-3c69ceb9286a -i td/test-erasure-eio/3/new.json 2024-09-15T02:34:47.773 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:34:47.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:34:47.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHR+ZmAQvUBhAA4muikTvXtIA9nY2WgDrqmQ== --osd-uuid 0dae3d3c-a54a-4e42-8a56-3c69ceb9286a 2024-09-15T02:34:47.809 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:47.808+0000 7f61a0f35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:47.811 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:47.810+0000 7f61a0f35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:47.811 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:47.811+0000 7f61a0f35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:47.812 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:47.811+0000 7f61a0f35440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:34:47.812 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:47.811+0000 7f61a0f35440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:34:47.812 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:47.811+0000 7f61a0f35440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:34:47.906 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:47.905+0000 7f61a0f35440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:34:53.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:34:53.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:34:53.842 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:34:53.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:34:53.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:34:54.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:34:54.325 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:34:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:34:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:34:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:34:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:34:54.348 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:54.347+0000 7f22dd28d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:54.349 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:54.349+0000 7f22dd28d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:54.351 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:54.350+0000 7f22dd28d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:34:54.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:34:54.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:34:54.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:34:54.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:34:54.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:34:54.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:54.676 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:34:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:34:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:54.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:34:54.868 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:54.867+0000 7f22dd28d440 -1 Falling back to public interface 2024-09-15T02:34:55.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:56.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:56.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:56.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:34:56.038 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:34:56.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:56.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:34:56.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:57.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:57.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:57.392 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:34:57.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:34:57.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:57.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:34:57.467 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:34:57.465+0000 7f22dd28d440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:34:57.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:34:58.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:34:58.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:34:58.784 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:34:58.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:34:58.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:34:58.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:34:59.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:00.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:00.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:00.311 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:35:00.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:35:00.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:00.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:00.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:01.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:01.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:01.679 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:35:01.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:35:01.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:01.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:02.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:03.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:03.047 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:35:03.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:03.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:35:03.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:03.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:03.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:04.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:04.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:04.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:35:04.489 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:35:04.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:04.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:04.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:05.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:05.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:05.867 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:35:05.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:35:05.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:05.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:06.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:07.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:07.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:07.274 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:35:07.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:35:07.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:07.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:07.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:08.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:08.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:08.703 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:35:08.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:35:08.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:08.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:09.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:10.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:10.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:10.111 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:35:10.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:35:10.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:10.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:10.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:11.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:11.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:11.967 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:35:11.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:35:11.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:11.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:12.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:13.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:13.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:13.731 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:35:13.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:35:13.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:13.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:14.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:15.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:15.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:15.483 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:35:15.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:35:15.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:15.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:16.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:17.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:17.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:17.203 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:35:17.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:35:17.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:17.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:17.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:18.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:18.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:18.973 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:35:18.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:35:18.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:18.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:20.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:21.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:21.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:21.146 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:35:21.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:35:21.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:21.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:35:21.505 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/133140,v1:127.0.0.1:6827/133140] [v2:127.0.0.1:6828/133140,v1:127.0.0.1:6829/133140] exists,up 0dae3d3c-a54a-4e42-8a56-3c69ceb9286a 2024-09-15T02:35:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:35:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:35:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:35:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:35:21.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T02:35:21.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:35:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:35:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:35:21.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:35:21.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:35:21.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:35:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:35:21.515 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:35:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:35:21.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:35:21.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:35:21.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T02:35:21.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:35:21.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=94443985-1202-43ef-9048-1b9af1308dfc 2024-09-15T02:35:21.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 94443985-1202-43ef-9048-1b9af1308dfc' 2024-09-15T02:35:21.517 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 94443985-1202-43ef-9048-1b9af1308dfc 2024-09-15T02:35:21.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:35:21.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpR+ZmgVofHxAA4uKnc9j7okoGg3a7AEfSpQ== 2024-09-15T02:35:21.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpR+ZmgVofHxAA4uKnc9j7okoGg3a7AEfSpQ=="}' 2024-09-15T02:35:21.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 94443985-1202-43ef-9048-1b9af1308dfc -i td/test-erasure-eio/4/new.json 2024-09-15T02:35:22.098 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:35:22.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T02:35:22.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpR+ZmgVofHxAA4uKnc9j7okoGg3a7AEfSpQ== --osd-uuid 94443985-1202-43ef-9048-1b9af1308dfc 2024-09-15T02:35:22.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:22.132+0000 7f87d6d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:22.274 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:22.272+0000 7f87d6d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:22.276 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:22.273+0000 7f87d6d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:22.276 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:22.274+0000 7f87d6d23440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:35:22.276 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:22.274+0000 7f87d6d23440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:35:22.276 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:22.274+0000 7f87d6d23440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:35:22.382 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:22.380+0000 7f87d6d23440 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T02:35:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T02:35:29.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:35:29.451 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:35:29.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:35:29.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:35:29.932 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:35:29.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:35:29.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:35:29.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:35:29.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:35:29.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:35:29.955 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:29.954+0000 7f59bad25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:29.957 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:29.955+0000 7f59bad25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:29.958 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:29.956+0000 7f59bad25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:30.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:35:30.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:35:30.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:35:30.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:35:30.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:35:30.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:30.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:35:30.284 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:35:30.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:30.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:30.474 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:30.472+0000 7f59bad25440 -1 Falling back to public interface 2024-09-15T02:35:30.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:31.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:31.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:31.633 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:35:31.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:35:31.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:31.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:31.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:32.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:32.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:32.990 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:35:32.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:35:32.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:32.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:33.127 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:33.126+0000 7f59bad25440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:35:33.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:34.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:34.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:34.347 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:35:34.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:35:34.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:34.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:34.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:35.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:35.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:35.821 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:35:35.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:35:35.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:35.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:36.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:37.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:37.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:37.441 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:35:37.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:35:37.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:37.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:37.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:38.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:38.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:38.895 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:35:38.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:35:38.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:38.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:39.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:40.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:40.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:40.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:35:40.302 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:35:40.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:40.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:40.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:41.680 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:35:41.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:35:41.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:41.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:42.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:43.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:43.048 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:35:43.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:43.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:35:43.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:43.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:43.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:44.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:44.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:44.470 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:35:44.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:35:44.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:44.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:44.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:45.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:45.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:45.920 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:35:45.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:35:45.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:45.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:46.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:47.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:47.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:47.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:35:47.466 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:35:47.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:47.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:48.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:49.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:49.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:49.252 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:35:49.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:35:49.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:49.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:50.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:51.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:51.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:51.083 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:35:51.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:35:51.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:51.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:51.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:52.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:52.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:52.901 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:35:52.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:35:52.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:52.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:53.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:54.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:54.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:54.540 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:35:54.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:35:54.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:54.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:55.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:56.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:56.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:56.249 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:35:56.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:35:56.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:56.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:56.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:35:57.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:35:57.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:35:57.992 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:35:57.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:35:57.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:35:57.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:35:58.635 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/134894,v1:127.0.0.1:6835/134894] [v2:127.0.0.1:6836/134894,v1:127.0.0.1:6837/134894] exists,up 94443985-1202-43ef-9048-1b9af1308dfc 2024-09-15T02:35:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:35:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:35:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:35:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:35:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T02:35:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:35:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:35:58.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=935f07a1-52cc-4d85-82ef-e992a04712f6 2024-09-15T02:35:58.642 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 935f07a1-52cc-4d85-82ef-e992a04712f6 2024-09-15T02:35:58.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 935f07a1-52cc-4d85-82ef-e992a04712f6' 2024-09-15T02:35:58.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:35:58.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOSOZm7CfiJhAAVpbo6Z8Q3b5z+NSE2tCs9w== 2024-09-15T02:35:58.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOSOZm7CfiJhAAVpbo6Z8Q3b5z+NSE2tCs9w=="}' 2024-09-15T02:35:58.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 935f07a1-52cc-4d85-82ef-e992a04712f6 -i td/test-erasure-eio/5/new.json 2024-09-15T02:35:59.486 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:35:59.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T02:35:59.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOSOZm7CfiJhAAVpbo6Z8Q3b5z+NSE2tCs9w== --osd-uuid 935f07a1-52cc-4d85-82ef-e992a04712f6 2024-09-15T02:35:59.522 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:59.521+0000 7f417997a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:59.524 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:59.523+0000 7f417997a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:59.525 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:59.524+0000 7f417997a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:35:59.526 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:59.524+0000 7f417997a440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:35:59.526 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:59.524+0000 7f417997a440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:35:59.526 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:59.524+0000 7f417997a440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:35:59.659 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:35:59.657+0000 7f417997a440 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T02:36:05.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T02:36:05.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:36:05.864 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-15T02:36:05.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T02:36:05.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:36:06.349 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-15T02:36:06.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T02:36:06.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:36:06.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:36:06.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:36:06.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:36:06.371 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:06.370+0000 7f621f008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:06.376 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:06.372+0000 7f621f008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:06.377 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:06.373+0000 7f621f008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:06.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T02:36:06.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:36:06.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T02:36:06.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:36:06.747 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:36:06.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:36:06.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:06.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:36:06.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:06.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:06.896 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:06.895+0000 7f621f008440 -1 Falling back to public interface 2024-09-15T02:36:07.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:08.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:08.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:08.096 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:36:08.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:36:08.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:08.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:08.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:09.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:09.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:09.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:36:09.840 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:36:09.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:09.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:09.840 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:09.486+0000 7f621f008440 -1 osd.5 0 log_to_monitors true 2024-09-15T02:36:09.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:10.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:10.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:10.939 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:36:10.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:36:10.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:10.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:11.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:12.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:12.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:12.674 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:36:12.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:36:12.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:12.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:13.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:14.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:14.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:14.150 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:36:14.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:36:14.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:14.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:14.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:15.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:15.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:15.623 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:36:15.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:36:15.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:15.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:15.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:16.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:16.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:16.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:36:16.988 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:36:16.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:16.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:17.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:18.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:18.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:18.455 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:36:18.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:36:18.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:18.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:18.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:19.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:19.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:19.847 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:36:19.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:36:19.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:19.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:20.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:21.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:21.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:21.251 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:36:21.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:36:21.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:21.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:21.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:22.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:22.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:22.973 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:36:22.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:36:22.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:22.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:24.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:25.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:25.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:25.070 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:36:25.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:36:25.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:25.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:25.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:26.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:26.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:26.470 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:36:26.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:36:26.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:26.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:26.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:27.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:27.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:27.989 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:36:27.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:36:27.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:27.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:28.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:29.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:29.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:29.970 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:36:29.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:36:29.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:29.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:30.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:31.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:31.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:31.740 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:36:31.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:36:31.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:31.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:32.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:33.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:33.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:33.351 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:36:33.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:36:33.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:33.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:36:33.743 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/136673,v1:127.0.0.1:6843/136673] [v2:127.0.0.1:6844/136673,v1:127.0.0.1:6845/136673] exists,up 935f07a1-52cc-4d85-82ef-e992a04712f6 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:36:33.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:36:33.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:36:33.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47219c31-2a4d-4160-bb4b-997d67e3290b 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 47219c31-2a4d-4160-bb4b-997d67e3290b' 2024-09-15T02:36:33.750 INFO:tasks.workunit.client.0.smithi130.stdout:add osd6 47219c31-2a4d-4160-bb4b-997d67e3290b 2024-09-15T02:36:33.751 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:36:33.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxSOZm1TRFLRAA4el2N//g1EBA8IdDKD8V2g== 2024-09-15T02:36:33.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxSOZm1TRFLRAA4el2N//g1EBA8IdDKD8V2g=="}' 2024-09-15T02:36:33.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47219c31-2a4d-4160-bb4b-997d67e3290b -i td/test-erasure-eio/6/new.json 2024-09-15T02:36:34.713 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:36:34.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T02:36:34.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxSOZm1TRFLRAA4el2N//g1EBA8IdDKD8V2g== --osd-uuid 47219c31-2a4d-4160-bb4b-997d67e3290b 2024-09-15T02:36:34.749 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:34.748+0000 7f2781027440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:34.751 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:34.750+0000 7f2781027440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:34.752 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:34.751+0000 7f2781027440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:34.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:34.751+0000 7f2781027440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:36:34.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:34.751+0000 7f2781027440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:36:34.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:34.751+0000 7f2781027440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:36:34.831 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:34.830+0000 7f2781027440 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T02:36:40.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T02:36:40.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:36:40.912 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd6 key to auth repository 2024-09-15T02:36:40.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-15T02:36:40.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:36:41.420 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.6 2024-09-15T02:36:41.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-15T02:36:41.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:36:41.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:36:41.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:36:41.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:36:41.444 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:41.442+0000 7fc8d120c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:41.467 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:41.466+0000 7fc8d120c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:41.468 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:41.467+0000 7fc8d120c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:36:41.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-15T02:36:41.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:36:41.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-15T02:36:41.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:36:41.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:36:41.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:41.814 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:36:41.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:36:41.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:41.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:41.998 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:41.997+0000 7fc8d120c440 -1 Falling back to public interface 2024-09-15T02:36:42.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:43.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:43.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:43.183 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:36:43.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:36:43.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:43.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:43.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:44.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:44.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:44.541 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:36:44.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:36:44.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:44.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:44.565 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:36:44.564+0000 7fc8d120c440 -1 osd.6 0 log_to_monitors true 2024-09-15T02:36:45.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:46.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:46.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:46.011 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:36:46.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:36:46.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:46.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:46.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:47.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:47.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:47.415 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:36:47.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:36:47.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:47.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:47.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:48.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:48.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:48.824 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:36:48.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:36:48.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:48.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:49.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:50.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:50.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:50.298 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:36:50.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:36:50.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:50.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:50.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:51.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:51.705 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:36:51.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:51.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:36:51.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:51.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:52.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:53.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:53.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:53.096 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:36:53.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:36:53.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:53.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:53.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:54.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:54.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:54.586 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:36:54.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:36:54.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:54.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:54.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:55.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:55.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:55.949 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:36:55.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:36:55.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:55.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:56.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:57.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:57.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:57.416 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:36:57.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:36:57.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:57.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:57.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:36:58.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:36:58.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:36:58.786 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:36:58.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:36:58.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:36:58.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:36:59.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:00.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:00.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:00.691 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:37:00.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:37:00.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:00.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:37:01.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:02.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:02.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:02.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:37:02.796 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:37:02.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:02.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:37:03.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:04.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:04.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:04.370 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:37:04.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:37:04.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:04.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:37:04.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:05.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:05.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:05.800 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:37:05.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:37:05.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:05.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:37:06.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:07.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:07.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:07.510 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:37:07.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:37:07.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:07.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:37:08.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:09.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:09.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:09.058 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:37:09.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:37:09.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:09.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:37:09.428 INFO:tasks.workunit.client.0.smithi130.stdout:osd.6 up in weight 1 up_from 62 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/138427,v1:127.0.0.1:6851/138427] [v2:127.0.0.1:6852/138427,v1:127.0.0.1:6853/138427] exists,up 47219c31-2a4d-4160-bb4b-997d67e3290b 2024-09-15T02:37:09.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:37:09.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:37:09.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:37:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2024-09-15T02:37:09.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:37:09.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a393c4e-ba1c-4500-b227-3e23d7624726 2024-09-15T02:37:09.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 1a393c4e-ba1c-4500-b227-3e23d7624726' 2024-09-15T02:37:09.436 INFO:tasks.workunit.client.0.smithi130.stdout:add osd7 1a393c4e-ba1c-4500-b227-3e23d7624726 2024-09-15T02:37:09.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:37:09.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVSOZmJ3ucGhAAinKQoyjystTopXzc7rVi4w== 2024-09-15T02:37:09.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVSOZmJ3ucGhAAinKQoyjystTopXzc7rVi4w=="}' 2024-09-15T02:37:09.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a393c4e-ba1c-4500-b227-3e23d7624726 -i td/test-erasure-eio/7/new.json 2024-09-15T02:37:10.429 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:37:10.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2024-09-15T02:37:10.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVSOZmJ3ucGhAAinKQoyjystTopXzc7rVi4w== --osd-uuid 1a393c4e-ba1c-4500-b227-3e23d7624726 2024-09-15T02:37:10.464 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:10.463+0000 7fec60e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:10.466 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:10.465+0000 7fec60e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:10.467 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:10.466+0000 7fec60e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:10.467 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:10.466+0000 7fec60e12440 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-15T02:37:10.467 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:10.466+0000 7fec60e12440 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-15T02:37:10.468 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:10.466+0000 7fec60e12440 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-15T02:37:10.570 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:10.569+0000 7fec60e12440 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-09-15T02:37:16.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-09-15T02:37:16.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:37:16.428 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd7 key to auth repository 2024-09-15T02:37:16.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-15T02:37:16.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:37:16.913 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.7 2024-09-15T02:37:16.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-15T02:37:16.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:37:16.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:37:16.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:37:16.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:37:16.935 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:16.933+0000 7fe51a40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:16.938 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:16.935+0000 7fe51a40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:16.938 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:16.936+0000 7fe51a40c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:17.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-15T02:37:17.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:37:17.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-15T02:37:17.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:37:17.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:37:17.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:17.265 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:37:17.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:37:17.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:17.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:17.457 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:17.455+0000 7fe51a40c440 -1 Falling back to public interface 2024-09-15T02:37:17.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:18.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:18.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:18.625 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:37:18.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:37:18.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:18.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:19.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:19.957 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:19.954+0000 7fe51a40c440 -1 osd.7 0 log_to_monitors true 2024-09-15T02:37:20.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:20.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:20.004 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:37:20.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:37:20.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:20.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:20.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:21.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:21.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:21.739 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:37:21.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:37:21.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:21.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:22.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:23.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:23.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:23.198 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:37:23.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:37:23.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:23.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:23.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:24.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:24.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:24.562 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:37:24.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:37:24.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:24.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:24.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:25.951 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:37:25.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:37:25.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:25.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:26.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:27.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:27.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:27.350 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:37:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:37:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:27.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:28.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:28.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:28.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:37:28.833 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:37:28.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:28.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:29.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:30.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:30.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:30.290 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:37:30.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:37:30.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:30.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:30.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:31.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:31.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:31.663 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:37:31.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:37:31.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:31.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:32.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:33.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:33.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:33.074 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:37:33.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:37:33.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:33.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:33.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:34.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:34.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:34.725 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:37:34.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:37:34.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:34.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:36.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:36.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:36.723 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:37:36.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:37:36.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:36.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:37.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:38.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:38.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:38.921 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:37:38.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:37:38.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:38.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:39.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:40.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:40.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:40.825 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:37:40.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:37:40.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:40.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:42.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:42.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:42.637 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:37:42.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:37:42.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:42.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:43.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:44.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:44.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:44.335 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:37:44.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:37:44.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:44.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:44.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:45.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:45.843 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:37:45.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:45.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:37:45.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:45.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-15T02:37:46.384 INFO:tasks.workunit.client.0.smithi130.stdout:osd.7 up in weight 1 up_from 71 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/140206,v1:127.0.0.1:6859/140206] [v2:127.0.0.1:6860/140206,v1:127.0.0.1:6861/140206] exists,up 1a393c4e-ba1c-4500-b227-3e23d7624726 2024-09-15T02:37:46.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:37:46.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:37:46.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:37:46.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:37:46.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:37:46.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:37:46.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c2dce89-9168-486b-96e2-bb4f0ea981c5 2024-09-15T02:37:46.391 INFO:tasks.workunit.client.0.smithi130.stdout:add osd8 6c2dce89-9168-486b-96e2-bb4f0ea981c5 2024-09-15T02:37:46.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 6c2dce89-9168-486b-96e2-bb4f0ea981c5' 2024-09-15T02:37:46.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:37:46.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6SOZmV8XqFxAAsW/n6cS7yBiTAQRevDqj6g== 2024-09-15T02:37:46.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6SOZmV8XqFxAAsW/n6cS7yBiTAQRevDqj6g=="}' 2024-09-15T02:37:46.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c2dce89-9168-486b-96e2-bb4f0ea981c5 -i td/test-erasure-eio/8/new.json 2024-09-15T02:37:47.305 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:37:47.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2024-09-15T02:37:47.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6SOZmV8XqFxAAsW/n6cS7yBiTAQRevDqj6g== --osd-uuid 6c2dce89-9168-486b-96e2-bb4f0ea981c5 2024-09-15T02:37:47.340 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:47.339+0000 7f6221a39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:47.342 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:47.341+0000 7f6221a39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:47.342+0000 7f6221a39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:47.342+0000 7f6221a39440 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-15T02:37:47.344 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:47.342+0000 7f6221a39440 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-15T02:37:47.344 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:47.342+0000 7f6221a39440 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-15T02:37:47.457 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:47.456+0000 7f6221a39440 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-09-15T02:37:53.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-09-15T02:37:53.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:37:53.653 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd8 key to auth repository 2024-09-15T02:37:53.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-09-15T02:37:53.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:37:54.132 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.8 2024-09-15T02:37:54.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-09-15T02:37:54.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:37:54.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:37:54.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:37:54.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:37:54.154 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:54.153+0000 7fe8b551c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:54.179 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:54.178+0000 7fe8b551c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:54.180 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:54.179+0000 7fe8b551c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:37:54.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-09-15T02:37:54.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:37:54.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-15T02:37:54.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:37:54.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:37:54.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:54.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:37:54.520 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:37:54.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:54.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:37:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:54.711+0000 7fe8b551c440 -1 Falling back to public interface 2024-09-15T02:37:54.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:55.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:55.887 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:37:55.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:37:55.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:55.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:37:56.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:57.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:57.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:57.246 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:37:57.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:37:57.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:57.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:37:57.293 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:37:57.292+0000 7fe8b551c440 -1 osd.8 0 log_to_monitors true 2024-09-15T02:37:57.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:37:58.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:37:58.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:37:58.610 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:37:58.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:37:58.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:37:58.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:37:59.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:00.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:00.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:00.065 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:38:00.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:38:00.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:00.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:00.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:01.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:01.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:01.563 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:38:01.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:38:01.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:01.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:01.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:02.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:02.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:02.942 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:38:02.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:38:02.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:02.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:03.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:04.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:04.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:04.418 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:38:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:38:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:04.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:04.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:05.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:05.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:05.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:38:05.787 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:38:05.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:05.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:06.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:07.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:07.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:07.195 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:38:07.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:38:07.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:07.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:07.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:08.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:08.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:08.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:38:08.568 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:38:08.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:08.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:09.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:10.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:10.168 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:38:10.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:10.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:38:10.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:10.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:10.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:11.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:11.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:11.579 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:38:11.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:38:11.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:11.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:13.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:13.495 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:38:13.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:13.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:38:13.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:13.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:14.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:15.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:15.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:15.519 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:38:15.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:38:15.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:15.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:16.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:17.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:17.375 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:38:17.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:17.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:38:17.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:17.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:18.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:19.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:19.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:19.129 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:38:19.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:38:19.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:19.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:19.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:20.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:20.893 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:38:20.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:38:20.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:20.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:21.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:22.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:22.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:22.463 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:38:22.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:38:22.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:22.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:22.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:23.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:23.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:23.852 INFO:tasks.workunit.client.0.smithi130.stdout:19 2024-09-15T02:38:23.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T02:38:23.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:23.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-15T02:38:24.369 INFO:tasks.workunit.client.0.smithi130.stdout:osd.8 up in weight 1 up_from 82 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/141985,v1:127.0.0.1:6867/141985] [v2:127.0.0.1:6868/141985,v1:127.0.0.1:6869/141985] exists,up 6c2dce89-9168-486b-96e2-bb4f0ea981c5 2024-09-15T02:38:24.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:38:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:38:24.445 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:38:24.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:38:24.458 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:33:13.575+0000 7f1020d21440 0 load: jerasure load: lrc 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-09-15T02:38:24.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:38:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-09-15T02:38:25.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:38:25.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:38:26.252 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:38:26.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:38:27.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:38:27.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:38:27.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:38:27.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:38:27.265 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:38:27.266 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:38:27.266 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:38:27.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:38:27.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:38:27.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:38:27.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:38:27.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:38:27.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:38:27.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:38:27.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:38:27.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:38:27.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:38:27.751 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:8' 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:27.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:38:27.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803843 2024-09-15T02:38:27.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803843 2024-09-15T02:38:27.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843' 2024-09-15T02:38:27.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:27.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:38:28.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574906 2024-09-15T02:38:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574906 2024-09-15T02:38:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906' 2024-09-15T02:38:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:28.075 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:38:28.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378676 2024-09-15T02:38:28.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378676 2024-09-15T02:38:28.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906 2-81604378676' 2024-09-15T02:38:28.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:28.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:38:28.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084332 2024-09-15T02:38:28.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084332 2024-09-15T02:38:28.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906 2-81604378676 3-120259084332' 2024-09-15T02:38:28.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:28.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:38:28.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659172 2024-09-15T02:38:28.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659172 2024-09-15T02:38:28.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906 2-81604378676 3-120259084332 4-176093659172' 2024-09-15T02:38:28.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:28.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:38:28.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332124 2024-09-15T02:38:28.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332124 2024-09-15T02:38:28.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906 2-81604378676 3-120259084332 4-176093659172 5-219043332124' 2024-09-15T02:38:28.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:28.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:38:29.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=266287972372 2024-09-15T02:38:29.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 266287972372 2024-09-15T02:38:29.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906 2-81604378676 3-120259084332 4-176093659172 5-219043332124 6-266287972372' 2024-09-15T02:38:29.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:29.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:38:29.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=304942678030 2024-09-15T02:38:29.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 304942678030 2024-09-15T02:38:29.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906 2-81604378676 3-120259084332 4-176093659172 5-219043332124 6-266287972372 7-304942678030' 2024-09-15T02:38:29.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:29.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:38:29.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318278 2024-09-15T02:38:29.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318278 2024-09-15T02:38:29.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803843 1-55834574906 2-81604378676 3-120259084332 4-176093659172 5-219043332124 6-266287972372 7-304942678030 8-352187318278' 2024-09-15T02:38:29.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:29.221 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803843 2024-09-15T02:38:29.221 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:29.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:38:29.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803843 2024-09-15T02:38:29.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:29.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803843 2024-09-15T02:38:29.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803843' 2024-09-15T02:38:29.223 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803843 2024-09-15T02:38:29.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:38:29.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803843 -lt 25769803843 2024-09-15T02:38:29.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:29.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574906 2024-09-15T02:38:29.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:29.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:38:29.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574906 2024-09-15T02:38:29.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:29.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574906 2024-09-15T02:38:29.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574906' 2024-09-15T02:38:29.616 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574906 2024-09-15T02:38:29.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:38:29.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574906 -lt 55834574906 2024-09-15T02:38:29.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:29.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378676 2024-09-15T02:38:29.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:29.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:38:29.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378676 2024-09-15T02:38:29.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:29.969 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378676 2024-09-15T02:38:29.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378676 2024-09-15T02:38:29.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378676' 2024-09-15T02:38:29.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:38:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378676 -lt 81604378676 2024-09-15T02:38:30.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:30.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084332 2024-09-15T02:38:30.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:30.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:38:30.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084332 2024-09-15T02:38:30.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:30.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084332 2024-09-15T02:38:30.344 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084332 2024-09-15T02:38:30.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084332' 2024-09-15T02:38:30.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:38:30.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084332 -lt 120259084332 2024-09-15T02:38:30.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:30.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-176093659172 2024-09-15T02:38:30.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:30.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:38:30.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-176093659172 2024-09-15T02:38:30.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:30.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659172 2024-09-15T02:38:30.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 176093659172' 2024-09-15T02:38:30.760 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 176093659172 2024-09-15T02:38:30.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:38:31.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659172 -lt 176093659172 2024-09-15T02:38:31.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:31.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-219043332124 2024-09-15T02:38:31.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:31.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:38:31.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-219043332124 2024-09-15T02:38:31.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:31.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332124 2024-09-15T02:38:31.131 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 219043332124 2024-09-15T02:38:31.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 219043332124' 2024-09-15T02:38:31.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:38:31.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332124 -lt 219043332124 2024-09-15T02:38:31.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:31.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-266287972372 2024-09-15T02:38:31.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:31.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:38:31.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-266287972372 2024-09-15T02:38:31.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:31.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=266287972372 2024-09-15T02:38:31.516 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 266287972372 2024-09-15T02:38:31.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 266287972372' 2024-09-15T02:38:31.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:38:31.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 266287972373 -lt 266287972372 2024-09-15T02:38:31.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:31.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-304942678030 2024-09-15T02:38:31.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:31.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-15T02:38:31.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-304942678030 2024-09-15T02:38:31.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:31.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=304942678030 2024-09-15T02:38:31.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 304942678030' 2024-09-15T02:38:31.956 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.7 seq 304942678030 2024-09-15T02:38:31.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-15T02:38:32.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 304942678030 -lt 304942678030 2024-09-15T02:38:32.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:32.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-352187318278 2024-09-15T02:38:32.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:32.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:38:32.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-352187318278 2024-09-15T02:38:32.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:32.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318278 2024-09-15T02:38:32.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 352187318278' 2024-09-15T02:38:32.312 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.8 seq 352187318278 2024-09-15T02:38:32.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:38:32.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318278 -lt 352187318278 2024-09-15T02:38:32.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:38:32.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:38:32.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:38:33.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:38:33.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:38:33.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:38:33.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:38:33.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:38:33.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:38:33.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:38:33.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:38:33.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:38:33.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:38:33.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:38:33.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:38:33.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:38:33.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:38:33.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:38:33.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:38:33.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:38:33.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T02:38:34.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-15T02:38:34.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:38:34.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:38:34.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:38:34.200 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:38:34.200 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:38:34.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:38:34.621 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:38:34.621 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:7' 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:38:34.622 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:38:34.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:38:34.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:38:34.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:38:34.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:38:34.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T02:38:34.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:38:34.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:38:34.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:34.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:34.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:34.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:38:34.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:38:34.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-15T02:38:34.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-09-15T02:38:34.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:38:34.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:38:34.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:38:34.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=3 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:38:34.785 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:38:35.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:38:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:38:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:38:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:38:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr:7' 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:38:35.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/0/type 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:38:35.166 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-15T02:38:35.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:38:35.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:38:35.167 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:35.167 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:35.167 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:35.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:38:35.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:38:35.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:38:35.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:38:35.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:38:35.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-15T02:38:35.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:38:35.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:38:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:35.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:38:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:38:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-09-15T02:38:35.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-09-15T02:38:35.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=4 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-15T02:38:35.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:38:35.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:38:35.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:38:35.331 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:7' 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T02:38:35.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T02:38:35.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:38:35.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=6 2024-09-15T02:38:35.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:38:35.724 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/6/type 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-09-15T02:38:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-09-15T02:38:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.6.asok 2024-09-15T02:38:35.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:38:35.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:38:35.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.6 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.6.asok 2024-09-15T02:38:35.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:38:35.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-09-15T02:38:35.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-09-15T02:38:35.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:38:35.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:38:35.887 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:38:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:38:36.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:38:36.236 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:38:36.236 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:38:36.236 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:7' 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:38:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:38:36.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:38:36.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:38:36.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:38:36.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-09-15T02:38:37.251 INFO:tasks.workunit.client.0.smithi130.stderr:osd.7 is already down. 2024-09-15T02:38:37.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-09-15T02:38:37.940 INFO:tasks.workunit.client.0.smithi130.stderr:osd.7 is already out. 2024-09-15T02:38:37.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-09-15T02:38:37.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:38:37.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:38:37.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:38:37.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:38:37.951 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:38:37.952 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:38:37.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:38:37.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:38:37.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:38:38.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:38:38.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:38:38.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:38:38.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:38:38.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:38:38.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:6 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:7 2024-09-15T02:38:38.614 INFO:tasks.workunit.client.0.smithi130.stderr:8' 2024-09-15T02:38:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:38:38.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:38.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:38:39.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803848 2024-09-15T02:38:39.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803848 2024-09-15T02:38:39.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848' 2024-09-15T02:38:39.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:39.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:38:39.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574910 2024-09-15T02:38:39.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574910 2024-09-15T02:38:39.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848 1-55834574910' 2024-09-15T02:38:39.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:39.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:38:39.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378680 2024-09-15T02:38:39.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378680 2024-09-15T02:38:39.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848 1-55834574910 2-81604378680' 2024-09-15T02:38:39.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:39.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:38:39.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084336 2024-09-15T02:38:39.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084336 2024-09-15T02:38:39.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848 1-55834574910 2-81604378680 3-120259084336' 2024-09-15T02:38:39.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:39.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:38:39.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659176 2024-09-15T02:38:39.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659176 2024-09-15T02:38:39.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848 1-55834574910 2-81604378680 3-120259084336 4-176093659176' 2024-09-15T02:38:39.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:39.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:38:39.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332128 2024-09-15T02:38:39.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332128 2024-09-15T02:38:39.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848 1-55834574910 2-81604378680 3-120259084336 4-176093659176 5-219043332128' 2024-09-15T02:38:39.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:39.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:38:39.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=266287972377 2024-09-15T02:38:39.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 266287972377 2024-09-15T02:38:39.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848 1-55834574910 2-81604378680 3-120259084336 4-176093659176 5-219043332128 6-266287972377' 2024-09-15T02:38:39.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:39.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-15T02:38:40.001 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-15T02:38:40.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T02:38:40.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T02:38:40.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T02:38:40.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:38:40.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-15T02:38:40.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318282 2024-09-15T02:38:40.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318282 2024-09-15T02:38:40.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803848 1-55834574910 2-81604378680 3-120259084336 4-176093659176 5-219043332128 6-266287972377 8-352187318282' 2024-09-15T02:38:40.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:40.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803848 2024-09-15T02:38:40.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:40.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:38:40.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803848 2024-09-15T02:38:40.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:40.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803848 2024-09-15T02:38:40.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803848' 2024-09-15T02:38:40.112 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803848 2024-09-15T02:38:40.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:38:40.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803845 -lt 25769803848 2024-09-15T02:38:40.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:38:41.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:38:41.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803848 -lt 25769803848 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574910 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574910 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:41.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574910 2024-09-15T02:38:41.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574910' 2024-09-15T02:38:41.826 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574910 2024-09-15T02:38:41.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:38:42.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574910 -lt 55834574910 2024-09-15T02:38:42.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:42.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378680 2024-09-15T02:38:42.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:42.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:38:42.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378680 2024-09-15T02:38:42.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:42.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378680 2024-09-15T02:38:42.189 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378680 2024-09-15T02:38:42.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378680' 2024-09-15T02:38:42.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:38:42.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378681 -lt 81604378680 2024-09-15T02:38:42.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:42.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084336 2024-09-15T02:38:42.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:42.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:38:42.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084336 2024-09-15T02:38:42.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:42.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084336 2024-09-15T02:38:42.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084336' 2024-09-15T02:38:42.583 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084336 2024-09-15T02:38:42.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:38:42.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084337 -lt 120259084336 2024-09-15T02:38:42.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:42.933 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-176093659176 2024-09-15T02:38:42.933 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:42.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:38:42.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-176093659176 2024-09-15T02:38:42.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659176 2024-09-15T02:38:42.936 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 176093659176 2024-09-15T02:38:42.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 176093659176' 2024-09-15T02:38:42.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:38:43.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659176 -lt 176093659176 2024-09-15T02:38:43.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:43.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-219043332128 2024-09-15T02:38:43.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:43.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:38:43.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-219043332128 2024-09-15T02:38:43.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:43.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332128 2024-09-15T02:38:43.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 219043332128' 2024-09-15T02:38:43.293 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 219043332128 2024-09-15T02:38:43.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:38:43.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332128 -lt 219043332128 2024-09-15T02:38:43.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:43.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-266287972377 2024-09-15T02:38:43.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:43.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:38:43.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-266287972377 2024-09-15T02:38:43.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:43.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=266287972377 2024-09-15T02:38:43.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 266287972377' 2024-09-15T02:38:43.644 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 266287972377 2024-09-15T02:38:43.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:38:44.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 266287972377 -lt 266287972377 2024-09-15T02:38:44.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:38:44.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-352187318282 2024-09-15T02:38:44.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:38:44.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-15T02:38:44.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-352187318282 2024-09-15T02:38:44.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:38:44.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318282 2024-09-15T02:38:44.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 352187318282' 2024-09-15T02:38:44.012 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.8 seq 352187318282 2024-09-15T02:38:44.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-15T02:38:44.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318282 -lt 352187318282 2024-09-15T02:38:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:38:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:38:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:38:44.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:38:45.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:38:45.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:38:45.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:38:45.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:38:45.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:38:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-15T02:38:45.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:38:45.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:38:45.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-09-15T02:38:45.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:38:45.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:38:46.301 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:38:46.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T02:38:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T02:38:47.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T02:38:47.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:38:47.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:38:47.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:38:47.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:38:47.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:38:47.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:38:47.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:38:47.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:38:47.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:38:47.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:38:47.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:38:47.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:38:47.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:38:47.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:38:47.539 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:38:47.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:38:47.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:38:47.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:38:47.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:38:47.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:38:47.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:38:47.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:38:47.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:38:47.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:47.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:47.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:38:47.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:38:47.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:38:47.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:38:47.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:38:47.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:38:47.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:38:47.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:38:47.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:38:47.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:38:47.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:38:47.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:38:47.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:38:47.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:38:47.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:38:47.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:38:47.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:38:47.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:38:47.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:38:47.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:38:47.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:38:47.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:38:47.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:47.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:47.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:38:47.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:38:47.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:38:47.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:38:47.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:38:47.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:47.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:47.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:38:47.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:38:47.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:38:47.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:38:47.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:38:47.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:38:47.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:38:47.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:38:47.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:38:47.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:38:47.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:38:47.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:38:47.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:38:47.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:38:47.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:38:47.972 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:38:47.972 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:47.972 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:47.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:38:47.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:38:48.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:38:48.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:38:48.216 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:48.216 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:48.216 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:48.216 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:38:48.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:38:48.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:38:48.290 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:48.291 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:48.291 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:48.291 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:38:48.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:38:48.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:38:48.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:38:48.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:38:48.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:38:48.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:38:48.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:38:48.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:38:48.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:38:49.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:38:49.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:38:49.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:38:49.201 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:38:49.201 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:49.201 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:49.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:38:49.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:38:49.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:38:49.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:38:49.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:38:49.450 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:38:49.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:38:50.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:38:50.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:38:50.529 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:38:50.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:38:50.538 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:38:47.998+0000 7f7bfd6c2900 0 load: jerasure load: lrc 2024-09-15T02:38:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-09-15T02:38:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-09-15T02:38:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-09-15T02:38:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:38:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:38:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-09-15T02:38:50.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T02:38:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:38:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T02:38:50.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:38:50.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:38:50.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:38:50.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:38:50.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b1113fa-086b-498f-8136-b9925a08a140 2024-09-15T02:38:50.547 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 4b1113fa-086b-498f-8136-b9925a08a140 2024-09-15T02:38:50.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4b1113fa-086b-498f-8136-b9925a08a140' 2024-09-15T02:38:50.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:38:50.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6SOZmivJ4IRAAU6R55dvYbSnAy4SWgXFy1g== 2024-09-15T02:38:50.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6SOZmivJ4IRAAU6R55dvYbSnAy4SWgXFy1g=="}' 2024-09-15T02:38:50.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b1113fa-086b-498f-8136-b9925a08a140 -i td/test-erasure-eio/0/new.json 2024-09-15T02:38:50.838 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:38:50.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:38:50.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6SOZmivJ4IRAAU6R55dvYbSnAy4SWgXFy1g== --osd-uuid 4b1113fa-086b-498f-8136-b9925a08a140 2024-09-15T02:38:50.877 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:50.876+0000 7fcb97010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:38:50.883 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:50.882+0000 7fcb97010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:38:50.884 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:50.883+0000 7fcb97010440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:38:50.884 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:50.883+0000 7fcb97010440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:38:50.885 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:50.883+0000 7fcb97010440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:38:50.885 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:50.883+0000 7fcb97010440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:38:50.932 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:50.931+0000 7fcb97010440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:38:56.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:38:56.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:38:56.963 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:38:56.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:38:56.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:38:57.455 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:38:57.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:38:57.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:38:57.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:38:57.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:38:57.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:38:57.482 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:57.481+0000 7fc217016440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:38:57.483 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:57.483+0000 7fc217016440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:38:57.485 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:57.484+0000 7fc217016440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:38:57.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:38:57.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:38:57.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:38:57.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:38:57.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:38:57.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:57.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:38:57.852 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:38:57.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:57.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:38:58.010 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:38:58.009+0000 7fc217016440 -1 Falling back to public interface 2024-09-15T02:38:58.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:38:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:38:59.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:38:59.209 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:38:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:38:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:38:59.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:38:59.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:00.492 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:00.490+0000 7fc217016440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:39:00.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:00.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:00.569 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:39:00.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:39:00.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:00.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:01.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:02.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:02.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:02.087 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:39:02.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:39:02.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:02.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:02.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:03.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:03.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:03.452 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:39:03.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:39:03.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:03.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:03.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:04.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:04.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:04.841 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:39:04.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:39:04.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:04.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:05.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:06.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:06.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:06.325 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:39:06.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:39:06.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:06.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:06.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:07.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:07.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:07.750 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:39:07.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:39:07.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:07.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:08.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:09.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:09.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:09.214 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:39:09.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:39:09.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:09.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:09.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:10.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:10.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:10.585 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:39:10.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:39:10.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:10.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:10.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:11.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:11.992 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:39:11.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:39:11.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:11.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:12.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:13.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:13.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:13.927 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:39:13.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:39:13.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:13.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:14.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:15.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:15.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:15.538 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:39:15.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:39:15.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:15.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:17.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:17.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:17.338 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:39:17.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:39:17.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:17.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:18.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:19.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:19.091 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:39:19.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:19.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:39:19.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:19.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:20.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:21.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:21.021 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:39:21.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:21.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:39:21.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:21.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:22.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:23.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:23.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:23.159 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:39:23.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:39:23.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:23.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:39:23.586 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/145692,v1:127.0.0.1:6803/145692] [v2:127.0.0.1:6804/145692,v1:127.0.0.1:6805/145692] exists,up 4b1113fa-086b-498f-8136-b9925a08a140 2024-09-15T02:39:23.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:39:23.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:39:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:39:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:39:23.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:39:23.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:39:23.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:39:23.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=34acd3a3-ea71-44e8-b4ce-780bc0e7b18a 2024-09-15T02:39:23.594 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 34acd3a3-ea71-44e8-b4ce-780bc0e7b18a 2024-09-15T02:39:23.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 34acd3a3-ea71-44e8-b4ce-780bc0e7b18a' 2024-09-15T02:39:23.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:39:23.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbSOZm9/snJBAAMAyNgE/ixd9gm1EWzJCazg== 2024-09-15T02:39:23.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbSOZm9/snJBAAMAyNgE/ixd9gm1EWzJCazg=="}' 2024-09-15T02:39:23.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 34acd3a3-ea71-44e8-b4ce-780bc0e7b18a -i td/test-erasure-eio/1/new.json 2024-09-15T02:39:24.087 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:39:24.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:39:24.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbSOZm9/snJBAAMAyNgE/ixd9gm1EWzJCazg== --osd-uuid 34acd3a3-ea71-44e8-b4ce-780bc0e7b18a 2024-09-15T02:39:24.127 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:24.126+0000 7ff507b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:24.129 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:24.128+0000 7ff507b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:24.130 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:24.129+0000 7ff507b27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:24.130 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:24.129+0000 7ff507b27440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:39:24.131 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:24.129+0000 7ff507b27440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:39:24.131 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:24.129+0000 7ff507b27440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:39:24.186 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:24.185+0000 7ff507b27440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:39:29.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:39:29.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:39:29.781 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:39:29.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:39:29.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:39:30.265 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:39:30.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:39:30.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:39:30.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:39:30.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:39:30.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:39:30.292 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:30.291+0000 7f97f0923440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:30.294 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:30.293+0000 7f97f0923440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:30.295 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:30.294+0000 7f97f0923440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:39:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:39:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:39:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:39:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:39:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:30.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:39:30.617 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:39:30.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:30.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:30.814 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:30.813+0000 7f97f0923440 -1 Falling back to public interface 2024-09-15T02:39:31.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:32.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:32.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:32.005 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:39:32.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:39:32.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:32.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:32.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:33.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:33.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:33.366 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:39:33.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:39:33.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:33.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:33.418 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:33.417+0000 7f97f0923440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:39:33.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:34.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:34.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:34.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:39:34.762 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:39:34.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:34.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:35.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:36.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:36.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:36.285 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:39:36.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:39:36.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:36.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:36.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:37.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:37.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:37.647 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:39:37.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:39:37.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:37.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:38.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:39.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:39.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:39.087 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:39:39.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:39:39.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:39.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:39.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:40.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:40.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:40.611 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:39:40.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:39:40.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:40.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:41.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:42.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:42.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:42.031 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:39:42.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:39:42.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:42.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:42.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:43.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:43.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:43.462 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:39:43.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:39:43.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:43.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:43.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:44.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:44.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:44.884 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:39:44.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:39:44.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:44.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:45.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:46.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:46.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:46.244 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:39:46.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:39:46.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:46.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:46.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:47.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:47.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:47.991 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:39:47.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:39:47.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:47.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:48.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:49.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:49.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:49.687 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:39:49.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:39:49.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:49.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:50.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:51.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:51.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:51.594 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:39:51.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:39:51.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:51.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:52.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:53.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:53.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:53.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:39:53.482 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:39:53.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:53.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:54.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:55.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:55.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:55.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:39:55.418 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:39:55.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:55.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:39:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:39:56.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:39:56.912 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:39:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:39:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:39:56.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:39:57.433 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/147421,v1:127.0.0.1:6811/147421] [v2:127.0.0.1:6812/147421,v1:127.0.0.1:6813/147421] exists,up 34acd3a3-ea71-44e8-b4ce-780bc0e7b18a 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:39:57.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:39:57.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:39:57.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13d18ef8-0a88-4612-93b8-bd58bf4aba1d 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 13d18ef8-0a88-4612-93b8-bd58bf4aba1d' 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 13d18ef8-0a88-4612-93b8-bd58bf4aba1d 2024-09-15T02:39:57.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:39:57.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9SOZmOML+GhAAk7pjOtpmvOSzpEAhKurWaA== 2024-09-15T02:39:57.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9SOZmOML+GhAAk7pjOtpmvOSzpEAhKurWaA=="}' 2024-09-15T02:39:57.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13d18ef8-0a88-4612-93b8-bd58bf4aba1d -i td/test-erasure-eio/2/new.json 2024-09-15T02:39:58.211 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:39:58.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:39:58.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9SOZmOML+GhAAk7pjOtpmvOSzpEAhKurWaA== --osd-uuid 13d18ef8-0a88-4612-93b8-bd58bf4aba1d 2024-09-15T02:39:58.251 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:58.250+0000 7fd62e869440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:58.253 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:58.252+0000 7fd62e869440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:58.254 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:58.253+0000 7fd62e869440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:39:58.254 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:58.253+0000 7fd62e869440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:39:58.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:58.253+0000 7fd62e869440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:39:58.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:58.253+0000 7fd62e869440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:39:58.293 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:39:58.292+0000 7fd62e869440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:40:04.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:40:04.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:40:04.225 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:40:04.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:40:04.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:40:04.704 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:40:04.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:40:04.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:40:04.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:40:04.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:40:04.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:40:04.731 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:04.729+0000 7f761884f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:04.733 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:04.731+0000 7f761884f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:04.734 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:04.732+0000 7f761884f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:05.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:40:05.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:40:05.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:40:05.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:40:05.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:40:05.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:05.087 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:40:05.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:40:05.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:05.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:05.248 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:05.246+0000 7f761884f440 -1 Falling back to public interface 2024-09-15T02:40:05.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:06.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:06.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:06.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:40:06.441 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:40:06.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:06.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:06.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:07.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:07.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:07.809 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:40:07.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:40:07.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:07.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:07.827 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:07.825+0000 7f761884f440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:40:08.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:09.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:09.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:09.251 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:40:09.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:40:09.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:09.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:09.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:10.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:10.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:10.693 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:40:10.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:40:10.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:10.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:11.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:12.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:12.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:12.099 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:40:12.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:40:12.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:12.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:12.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:13.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:13.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:13.574 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:40:13.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:40:13.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:13.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:13.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:14.943 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:40:14.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:40:14.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:14.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:15.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:16.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:16.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:16.340 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:40:16.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:40:16.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:16.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:16.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:17.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:17.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:17.704 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:40:17.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:40:17.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:17.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:18.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:19.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:19.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:19.084 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:40:19.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:40:19.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:19.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:19.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:20.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:20.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:20.652 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:40:20.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:40:20.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:20.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:21.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:22.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:22.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:22.114 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:40:22.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:40:22.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:22.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:22.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:23.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:23.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:23.852 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:40:23.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:40:23.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:23.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:24.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:25.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:25.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:25.588 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:40:25.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:40:25.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:25.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:26.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:27.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:27.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:27.349 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:40:27.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:40:27.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:27.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:28.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:29.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:29.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:29.043 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:40:29.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:40:29.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:29.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:29.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:30.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:30.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:30.781 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:40:30.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:40:30.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:30.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:40:31.190 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/149175,v1:127.0.0.1:6819/149175] [v2:127.0.0.1:6820/149175,v1:127.0.0.1:6821/149175] exists,up 13d18ef8-0a88-4612-93b8-bd58bf4aba1d 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:40:31.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:40:31.195 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:40:31.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:40:31.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:40:31.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:40:31.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:40:31.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2cb5f555-9111-434e-944c-fa874a91be46 2024-09-15T02:40:31.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2cb5f555-9111-434e-944c-fa874a91be46' 2024-09-15T02:40:31.197 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 2cb5f555-9111-434e-944c-fa874a91be46 2024-09-15T02:40:31.197 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:40:31.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfSeZm2J2MDBAAs8rmhUNWn88dA6/371prJw== 2024-09-15T02:40:31.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfSeZm2J2MDBAAs8rmhUNWn88dA6/371prJw=="}' 2024-09-15T02:40:31.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2cb5f555-9111-434e-944c-fa874a91be46 -i td/test-erasure-eio/3/new.json 2024-09-15T02:40:32.279 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:40:32.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:40:32.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfSeZm2J2MDBAAs8rmhUNWn88dA6/371prJw== --osd-uuid 2cb5f555-9111-434e-944c-fa874a91be46 2024-09-15T02:40:32.319 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:32.318+0000 7fbb9d817440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:32.321 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:32.320+0000 7fbb9d817440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:32.322 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:32.321+0000 7fbb9d817440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:32.322 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:32.321+0000 7fbb9d817440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:40:32.322 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:32.321+0000 7fbb9d817440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:40:32.323 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:32.321+0000 7fbb9d817440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:40:32.437 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:32.436+0000 7fbb9d817440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:40:38.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:40:38.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:40:38.543 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:40:38.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:40:38.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:40:39.030 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:40:39.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:40:39.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:40:39.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:40:39.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:40:39.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:40:39.058 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:39.057+0000 7fd7f471d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:39.094 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:39.093+0000 7fd7f471d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:39.095 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:39.094+0000 7fd7f471d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:40:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:40:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:40:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:40:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:40:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:40:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:39.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:40:39.402 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:40:39.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:39.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:39.619 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:39.618+0000 7fd7f471d440 -1 Falling back to public interface 2024-09-15T02:40:39.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:40.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:40.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:40.765 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:40:40.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:40:40.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:40.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:41.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:42.099 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:40:42.098+0000 7fd7f471d440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:40:42.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:42.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:42.126 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:40:42.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:40:42.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:42.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:42.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:43.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:43.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:43.581 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:40:43.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:40:43.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:43.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:44.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:45.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:45.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:45.032 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:40:45.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:40:45.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:45.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:45.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:46.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:46.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:46.591 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:40:46.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:40:46.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:46.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:47.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:48.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:48.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:48.054 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:40:48.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:40:48.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:48.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:48.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:49.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:49.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:49.436 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:40:49.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:40:49.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:49.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:49.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:50.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:50.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:50.922 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:40:50.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:40:50.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:50.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:51.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:52.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:52.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:52.321 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:40:52.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:40:52.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:52.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:52.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:53.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:53.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:53.702 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:40:53.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:40:53.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:53.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:55.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:55.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:55.141 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:40:55.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:40:55.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:55.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:55.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:56.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:56.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:56.627 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:40:56.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:40:56.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:56.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:57.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:40:58.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:40:58.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:40:58.413 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:40:58.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:40:58.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:40:58.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:40:59.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:00.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:00.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:00.189 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:41:00.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:41:00.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:00.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:41:01.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:02.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:02.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:02.136 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:41:02.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:41:02.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:02.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:41:02.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:03.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:03.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:03.748 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:41:03.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:41:03.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:03.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:41:04.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:05.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:05.450 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:41:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:41:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:41:06.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:07.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:07.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:07.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:41:07.140 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:41:07.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:07.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:41:07.574 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/150933,v1:127.0.0.1:6827/150933] [v2:127.0.0.1:6828/150933,v1:127.0.0.1:6829/150933] exists,up 2cb5f555-9111-434e-944c-fa874a91be46 2024-09-15T02:41:07.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:41:07.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:07.579 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:41:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:41:07.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:41:07.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T02:41:07.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:41:07.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a19d3a7-90f5-4933-8b73-12594d4a424f 2024-09-15T02:41:07.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 0a19d3a7-90f5-4933-8b73-12594d4a424f' 2024-09-15T02:41:07.581 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 0a19d3a7-90f5-4933-8b73-12594d4a424f 2024-09-15T02:41:07.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:41:07.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDSeZmiQllIxAAdV6WT8N1wDQ7xMwaditRgQ== 2024-09-15T02:41:07.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDSeZmiQllIxAAdV6WT8N1wDQ7xMwaditRgQ=="}' 2024-09-15T02:41:07.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a19d3a7-90f5-4933-8b73-12594d4a424f -i td/test-erasure-eio/4/new.json 2024-09-15T02:41:08.579 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:41:08.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T02:41:08.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDSeZmiQllIxAAdV6WT8N1wDQ7xMwaditRgQ== --osd-uuid 0a19d3a7-90f5-4933-8b73-12594d4a424f 2024-09-15T02:41:08.619 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:08.618+0000 7fb654e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:08.657 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:08.656+0000 7fb654e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:08.658 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:08.657+0000 7fb654e08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:08.658 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:08.657+0000 7fb654e08440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:41:08.658 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:08.657+0000 7fb654e08440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:41:08.658 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:08.657+0000 7fb654e08440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:41:08.780 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:08.778+0000 7fb654e08440 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T02:41:14.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T02:41:14.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:41:14.693 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:41:14.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:41:14.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:41:15.179 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:41:15.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:41:15.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:41:15.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:41:15.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:41:15.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:41:15.205 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:15.204+0000 7f535ec1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:15.206 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:15.206+0000 7f535ec1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:15.208 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:15.207+0000 7f535ec1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:15.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:41:15.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:41:15.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:41:15.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:41:15.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:41:15.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:15.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:41:15.575 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:41:15.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:15.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:15.735 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:15.734+0000 7f535ec1e440 -1 Falling back to public interface 2024-09-15T02:41:15.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:16.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:16.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:16.929 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:41:16.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:41:16.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:16.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:17.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:18.208 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:18.207+0000 7f535ec1e440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:41:18.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:18.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:18.290 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:41:18.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:41:18.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:18.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:18.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:19.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:19.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:19.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:41:19.995 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:41:19.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:19.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:20.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:21.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:21.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:21.652 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:41:21.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:41:21.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:21.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:22.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:23.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:23.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:23.053 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:41:23.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:41:23.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:23.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:23.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:24.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:24.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:24.517 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:41:24.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:41:24.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:24.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:25.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:26.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:26.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:26.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:41:26.088 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:41:26.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:26.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:26.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:27.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:27.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:27.454 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:41:27.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:41:27.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:27.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:28.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:29.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:29.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:29.027 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:41:29.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:41:29.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:29.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:29.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:30.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:30.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:30.526 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:41:30.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:41:30.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:30.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:30.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:31.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:31.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:31.967 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:41:31.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:41:31.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:31.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:32.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:33.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:33.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:33.707 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:41:33.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:41:33.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:33.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:34.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:35.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:35.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:35.443 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:41:35.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:41:35.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:35.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:36.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:37.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:37.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:37.335 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:41:37.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:41:37.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:37.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:37.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:38.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:38.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:38.916 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:41:38.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:41:38.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:38.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:39.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:40.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:40.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:40.975 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:41:40.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:41:40.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:40.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:41.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:42.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:42.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:42.522 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:41:42.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:41:42.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:42.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:41:43.197 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/152712,v1:127.0.0.1:6835/152712] [v2:127.0.0.1:6836/152712,v1:127.0.0.1:6837/152712] exists,up 0a19d3a7-90f5-4933-8b73-12594d4a424f 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:41:43.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:41:43.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:41:43.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:41:43.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:41:43.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:41:43.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:41:43.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:41:43.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:41:43.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:41:43.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:41:43.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:41:43.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T02:41:43.205 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:41:43.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=011816f1-8c63-44de-834f-b2038ad347b3 2024-09-15T02:41:43.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 011816f1-8c63-44de-834f-b2038ad347b3' 2024-09-15T02:41:43.206 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 011816f1-8c63-44de-834f-b2038ad347b3 2024-09-15T02:41:43.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:41:43.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnSeZmtg/tDBAAAfPkdoamuA0hizTFr6jwmA== 2024-09-15T02:41:43.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnSeZmtg/tDBAAAfPkdoamuA0hizTFr6jwmA=="}' 2024-09-15T02:41:43.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 011816f1-8c63-44de-834f-b2038ad347b3 -i td/test-erasure-eio/5/new.json 2024-09-15T02:41:43.994 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:41:44.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T02:41:44.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnSeZmtg/tDBAAAfPkdoamuA0hizTFr6jwmA== --osd-uuid 011816f1-8c63-44de-834f-b2038ad347b3 2024-09-15T02:41:44.034 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:44.033+0000 7f6bd5839440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:44.144 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:44.143+0000 7f6bd5839440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:44.145 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:44.144+0000 7f6bd5839440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:44.146 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:44.145+0000 7f6bd5839440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:41:44.146 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:44.145+0000 7f6bd5839440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:41:44.146 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:44.145+0000 7f6bd5839440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:41:44.368 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:44.367+0000 7f6bd5839440 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T02:41:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T02:41:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:41:50.517 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-15T02:41:50.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T02:41:50.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:41:51.018 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-15T02:41:51.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T02:41:51.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:41:51.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:41:51.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:41:51.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:41:51.045 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:51.044+0000 7f6864f84440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:51.047 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:51.046+0000 7f6864f84440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:51.048 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:51.047+0000 7f6864f84440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:41:51.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T02:41:51.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:41:51.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T02:41:51.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:41:51.407 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:41:51.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:41:51.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:51.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:41:51.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:51.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:41:51.566 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:51.565+0000 7f6864f84440 -1 Falling back to public interface 2024-09-15T02:41:51.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:52.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:52.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:52.800 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:41:52.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:41:52.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:52.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:41:53.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:54.072 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:41:54.070+0000 7f6864f84440 -1 osd.5 0 log_to_monitors true 2024-09-15T02:41:54.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:54.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:54.165 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:41:54.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:41:54.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:54.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:41:54.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:55.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:55.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:55.771 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:41:55.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:41:55.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:55.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:41:56.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:57.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:57.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:57.235 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:41:57.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:41:57.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:57.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:41:57.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:41:58.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:41:58.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:41:58.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:41:58.798 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:41:58.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:41:58.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:41:59.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:00.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:00.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:00.191 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:42:00.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:42:00.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:00.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:00.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:01.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:01.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:01.573 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:42:01.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:42:01.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:01.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:01.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:02.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:02.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:02.988 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:42:02.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:42:02.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:02.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:03.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:04.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:04.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:04.370 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:42:04.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:42:04.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:04.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:04.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:05.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:05.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:05.871 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:42:05.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:42:05.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:05.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:06.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:07.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:07.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:07.274 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:42:07.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:42:07.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:07.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:07.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:08.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:08.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:08.880 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:42:08.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:42:08.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:08.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:10.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:11.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:11.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:11.060 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:42:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:42:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:11.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:12.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:12.944 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:42:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:42:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:12.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:13.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:14.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:14.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:14.738 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:42:14.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:42:14.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:14.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:16.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:17.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:17.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:17.147 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:42:17.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:42:17.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:17.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:17.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:18.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:42:18.808 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:42:18.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:18.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:42:19.203 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/154467,v1:127.0.0.1:6843/154467] [v2:127.0.0.1:6844/154467,v1:127.0.0.1:6845/154467] exists,up 011816f1-8c63-44de-834f-b2038ad347b3 2024-09-15T02:42:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:42:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:42:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:42:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:42:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T02:42:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:42:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:42:19.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T02:42:19.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T02:42:19.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:42:19.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:42:19.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:42:19.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:42:19.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:42:19.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9ce234f-9c44-4e5d-8eb7-044c80e7e867 2024-09-15T02:42:19.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 e9ce234f-9c44-4e5d-8eb7-044c80e7e867' 2024-09-15T02:42:19.212 INFO:tasks.workunit.client.0.smithi130.stdout:add osd6 e9ce234f-9c44-4e5d-8eb7-044c80e7e867 2024-09-15T02:42:19.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:42:19.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLSeZmLxBMDRAAF4R3811eAt2KHmTdLBR8qA== 2024-09-15T02:42:19.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLSeZmLxBMDRAAF4R3811eAt2KHmTdLBR8qA=="}' 2024-09-15T02:42:19.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9ce234f-9c44-4e5d-8eb7-044c80e7e867 -i td/test-erasure-eio/6/new.json 2024-09-15T02:42:19.836 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:42:19.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T02:42:19.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLSeZmLxBMDRAAF4R3811eAt2KHmTdLBR8qA== --osd-uuid e9ce234f-9c44-4e5d-8eb7-044c80e7e867 2024-09-15T02:42:19.876 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:19.874+0000 7f9e5af35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:19.988 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:19.987+0000 7f9e5af35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:19.989 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:19.988+0000 7f9e5af35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:19.989 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:19.988+0000 7f9e5af35440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:42:19.990 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:19.989+0000 7f9e5af35440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:42:19.990 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:19.989+0000 7f9e5af35440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:42:20.105 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:20.104+0000 7f9e5af35440 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T02:42:26.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T02:42:26.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:42:26.921 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd6 key to auth repository 2024-09-15T02:42:26.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-15T02:42:26.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:42:27.406 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.6 2024-09-15T02:42:27.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-15T02:42:27.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:42:27.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:42:27.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:42:27.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:42:27.433 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:27.432+0000 7f0be5c3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:27.435 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:27.434+0000 7f0be5c3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:27.436 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:27.435+0000 7f0be5c3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:42:27.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-15T02:42:27.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:42:27.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-15T02:42:27.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:42:27.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:42:27.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:27.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:42:27.766 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:42:27.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:27.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:27.954 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:27.953+0000 7f0be5c3b440 -1 Falling back to public interface 2024-09-15T02:42:28.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:29.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:29.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:29.153 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:42:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:42:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:29.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:29.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:30.478 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:42:30.477+0000 7f0be5c3b440 -1 osd.6 0 log_to_monitors true 2024-09-15T02:42:30.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:30.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:30.522 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:42:30.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:42:30.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:30.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:31.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:32.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:32.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:32.055 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:42:32.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:42:32.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:32.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:32.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:33.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:33.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:33.647 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:42:33.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:42:33.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:33.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:34.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:35.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:35.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:35.115 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:42:35.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:42:35.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:35.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:35.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:36.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:36.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:36.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:42:36.590 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:42:36.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:36.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:37.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:38.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:38.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:38.045 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:42:38.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:42:38.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:38.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:38.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:39.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:39.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:39.439 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:42:39.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:42:39.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:39.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:39.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:40.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:40.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:40.894 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:42:40.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:42:40.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:40.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:41.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:42.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:42.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:42.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:42:42.341 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:42:42.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:42.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:42.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:43.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:43.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:43.739 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:42:43.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:42:43.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:43.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:44.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:45.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:45.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:45.342 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:42:45.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:42:45.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:45.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:46.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:47.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:47.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:47.313 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:42:47.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:42:47.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:47.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:48.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:49.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:49.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:49.370 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:42:49.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:42:49.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:49.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:49.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:42:50.903 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:42:50.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:50.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:51.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:52.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:52.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:52.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:42:52.329 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:42:52.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:52.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:53.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:54.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:54.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:54.019 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:42:54.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:42:54.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:54.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:54.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:42:55.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:42:55.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:42:55.404 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:42:55.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:42:55.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:42:55.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:42:55.822 INFO:tasks.workunit.client.0.smithi130.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/156221,v1:127.0.0.1:6851/156221] [v2:127.0.0.1:6852/156221,v1:127.0.0.1:6853/156221] exists,up e9ce234f-9c44-4e5d-8eb7-044c80e7e867 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:42:55.824 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:42:55.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:42:55.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:42:55.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:42:55.898 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:42:55.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:42:55.910 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:38:58.013+0000 7fc217016440 0 load: jerasure load: lrc 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:42:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T02:42:56.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:42:56.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:42:57.726 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:42:57.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:42:58.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:42:58.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:42:58.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:42:58.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:42:58.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:42:58.739 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:42:58.739 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:42:58.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:42:58.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:42:58.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:42:58.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:42:58.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:42:58.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:42:58.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:42:58.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:42:58.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:42:59.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:42:59.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:42:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803825 2024-09-15T02:42:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803825 2024-09-15T02:42:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825' 2024-09-15T02:42:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:42:59.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:42:59.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607595 2024-09-15T02:42:59.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607595 2024-09-15T02:42:59.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-51539607595' 2024-09-15T02:42:59.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:42:59.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:42:59.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=77309411370 2024-09-15T02:42:59.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 77309411370 2024-09-15T02:42:59.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-51539607595 2-77309411370' 2024-09-15T02:42:59.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:42:59.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:42:59.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084316 2024-09-15T02:42:59.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084316 2024-09-15T02:42:59.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-51539607595 2-77309411370 3-120259084316' 2024-09-15T02:42:59.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:42:59.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:42:59.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724573 2024-09-15T02:42:59.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724573 2024-09-15T02:42:59.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-51539607595 2-77309411370 3-120259084316 4-167503724573' 2024-09-15T02:42:59.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:42:59.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:43:00.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397518 2024-09-15T02:43:00.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397518 2024-09-15T02:43:00.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-51539607595 2-77309411370 3-120259084316 4-167503724573 5-210453397518' 2024-09-15T02:43:00.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:00.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:43:00.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037769 2024-09-15T02:43:00.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037769 2024-09-15T02:43:00.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-51539607595 2-77309411370 3-120259084316 4-167503724573 5-210453397518 6-257698037769' 2024-09-15T02:43:00.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:00.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803825 2024-09-15T02:43:00.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:00.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:43:00.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803825 2024-09-15T02:43:00.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:00.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803825 2024-09-15T02:43:00.183 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803825 2024-09-15T02:43:00.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-15T02:43:00.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:43:00.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803823 -lt 25769803825 2024-09-15T02:43:00.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:43:01.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:43:01.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:43:01.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803825 -lt 25769803825 2024-09-15T02:43:01.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:01.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607595 2024-09-15T02:43:01.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:01.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:43:01.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607595 2024-09-15T02:43:01.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:01.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607595 2024-09-15T02:43:01.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607595' 2024-09-15T02:43:01.889 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 51539607595 2024-09-15T02:43:01.890 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:43:02.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607595 -lt 51539607595 2024-09-15T02:43:02.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:02.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-77309411370 2024-09-15T02:43:02.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:02.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:43:02.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-77309411370 2024-09-15T02:43:02.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:02.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=77309411370 2024-09-15T02:43:02.238 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 77309411370 2024-09-15T02:43:02.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 77309411370' 2024-09-15T02:43:02.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:43:02.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 77309411371 -lt 77309411370 2024-09-15T02:43:02.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:02.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084316 2024-09-15T02:43:02.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:43:02.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084316 2024-09-15T02:43:02.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:02.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084316 2024-09-15T02:43:02.647 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084316 2024-09-15T02:43:02.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084316' 2024-09-15T02:43:02.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:43:02.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084317 -lt 120259084316 2024-09-15T02:43:02.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:02.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724573 2024-09-15T02:43:02.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:02.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:43:02.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724573 2024-09-15T02:43:02.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:02.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724573 2024-09-15T02:43:02.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724573' 2024-09-15T02:43:02.997 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 167503724573 2024-09-15T02:43:02.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:43:03.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724574 -lt 167503724573 2024-09-15T02:43:03.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:03.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-210453397518 2024-09-15T02:43:03.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:03.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:43:03.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-210453397518 2024-09-15T02:43:03.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:03.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397518 2024-09-15T02:43:03.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 210453397518' 2024-09-15T02:43:03.353 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 210453397518 2024-09-15T02:43:03.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397518 -lt 210453397518 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-257698037769 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-257698037769 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:03.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037769 2024-09-15T02:43:03.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 257698037769' 2024-09-15T02:43:03.710 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 257698037769 2024-09-15T02:43:03.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:43:04.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037769 -lt 257698037769 2024-09-15T02:43:04.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:43:04.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:43:04.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:43:04.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:43:04.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:43:04.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:43:04.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:43:04.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:43:05.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:43:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:43:05.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-15T02:43:05.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-15T02:43:05.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:43:05.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:43:05.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-15T02:43:05.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:43:05.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:43:05.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:43:05.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:43:05.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:43:05.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:43:05.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:43:05.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:43:05.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-15T02:43:06.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-09-15T02:43:06.603 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2024-09-15T02:43:06.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-09-15T02:43:06.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:43:06.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:43:06.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:43:06.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:43:06.616 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:43:06.616 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:43:06.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:43:06.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:43:06.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:43:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:43:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:43:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:43:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:43:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:43:06.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:43:07.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:07.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:43:07.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803828 2024-09-15T02:43:07.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803828 2024-09-15T02:43:07.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828' 2024-09-15T02:43:07.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:07.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:43:07.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607598 2024-09-15T02:43:07.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607598 2024-09-15T02:43:07.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-51539607598' 2024-09-15T02:43:07.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:07.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:43:07.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=77309411374 2024-09-15T02:43:07.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 77309411374 2024-09-15T02:43:07.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-51539607598 2-77309411374' 2024-09-15T02:43:07.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:07.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:43:08.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084320 2024-09-15T02:43:08.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084320 2024-09-15T02:43:08.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-51539607598 2-77309411374 3-120259084320' 2024-09-15T02:43:08.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:08.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:43:08.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724577 2024-09-15T02:43:08.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724577 2024-09-15T02:43:08.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-51539607598 2-77309411374 3-120259084320 4-167503724577' 2024-09-15T02:43:08.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:08.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:43:08.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397521 2024-09-15T02:43:08.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397521 2024-09-15T02:43:08.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-51539607598 2-77309411374 3-120259084320 4-167503724577 5-210453397521' 2024-09-15T02:43:08.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:43:08.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:43:08.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=257698037772 2024-09-15T02:43:08.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 257698037772 2024-09-15T02:43:08.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-51539607598 2-77309411374 3-120259084320 4-167503724577 5-210453397521 6-257698037772' 2024-09-15T02:43:08.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:08.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803828 2024-09-15T02:43:08.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:08.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:43:08.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803828 2024-09-15T02:43:08.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:08.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803828 2024-09-15T02:43:08.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-09-15T02:43:08.503 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803828 2024-09-15T02:43:08.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:43:08.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 25769803828 2024-09-15T02:43:08.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:43:09.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:43:09.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:43:10.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803829 -lt 25769803828 2024-09-15T02:43:10.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:10.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607598 2024-09-15T02:43:10.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:10.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:43:10.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607598 2024-09-15T02:43:10.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:10.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607598 2024-09-15T02:43:10.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607598' 2024-09-15T02:43:10.211 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 51539607598 2024-09-15T02:43:10.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:43:10.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607599 -lt 51539607598 2024-09-15T02:43:10.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:10.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-77309411374 2024-09-15T02:43:10.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:10.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:43:10.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-77309411374 2024-09-15T02:43:10.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:10.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=77309411374 2024-09-15T02:43:10.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 77309411374' 2024-09-15T02:43:10.562 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 77309411374 2024-09-15T02:43:10.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:43:10.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 77309411375 -lt 77309411374 2024-09-15T02:43:10.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:10.941 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084320 2024-09-15T02:43:10.941 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:10.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:43:10.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084320 2024-09-15T02:43:10.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:10.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084320 2024-09-15T02:43:10.943 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084320 2024-09-15T02:43:10.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084320' 2024-09-15T02:43:10.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:43:11.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084320 -lt 120259084320 2024-09-15T02:43:11.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:11.302 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724577 2024-09-15T02:43:11.302 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:11.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:43:11.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724577 2024-09-15T02:43:11.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:11.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724577 2024-09-15T02:43:11.305 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 167503724577 2024-09-15T02:43:11.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724577' 2024-09-15T02:43:11.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:43:11.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724577 -lt 167503724577 2024-09-15T02:43:11.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:11.655 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-210453397521 2024-09-15T02:43:11.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:11.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:43:11.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-210453397521 2024-09-15T02:43:11.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:11.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397521 2024-09-15T02:43:11.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 210453397521' 2024-09-15T02:43:11.658 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 210453397521 2024-09-15T02:43:11.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:43:12.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397522 -lt 210453397521 2024-09-15T02:43:12.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:43:12.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-257698037772 2024-09-15T02:43:12.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:43:12.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:43:12.006 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-257698037772 2024-09-15T02:43:12.006 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:43:12.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=257698037772 2024-09-15T02:43:12.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 257698037772' 2024-09-15T02:43:12.007 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 257698037772 2024-09-15T02:43:12.007 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:43:12.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 257698037773 -lt 257698037772 2024-09-15T02:43:12.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:43:12.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:43:12.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:43:12.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:43:13.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T02:43:13.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:43:13.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:43:13.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:43:13.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-15T02:43:13.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-15T02:43:13.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T02:43:13.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-15T02:43:13.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:43:13.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:43:13.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:43:13.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:43:13.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:43:13.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:43:13.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:43:13.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:43:13.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:43:13.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:43:14.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T02:43:14.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:43:14.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:43:14.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T02:43:14.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T02:43:15.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1535 2024-09-15T02:43:15.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1535 '!=' null 2024-09-15T02:43:15.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T02:43:15.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T02:43:15.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T02:43:15.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T02:43:15.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:43:15.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:43:15.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:43:15.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:43:15.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:43:15.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:43:15.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:43:15.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:43:15.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:43:15.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:43:15.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:43:15.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-15T02:43:16.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:43:16.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:43:16.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-15T02:43:16.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:43:16.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:43:16.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-15T02:43:16.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:43:16.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:43:16.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-15T02:43:16.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:43:16.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:43:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-15T02:43:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:43:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:43:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-15T02:43:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:43:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:43:16.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-15T02:43:16.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:43:16.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:43:16.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-09-15T02:43:16.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:43:16.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:43:16.748 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:43:16.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T02:43:17.458 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T02:43:17.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T02:43:17.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:43:17.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:43:17.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:43:17.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:43:17.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:43:17.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:43:17.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:43:17.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:43:17.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:43:17.580 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:43:17.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:43:17.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:43:17.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:43:17.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:43:17.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:43:17.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:17.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:43:17.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:43:17.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:17.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:43:17.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:43:17.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:43:17.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:43:17.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:17.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:17.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:43:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:43:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:43:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:43:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:43:17.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:43:17.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:43:17.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:43:17.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:43:17.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:43:17.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:43:17.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:43:17.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:43:17.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:17.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:43:17.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:43:17.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:43:17.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:43:17.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:43:17.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:43:17.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:43:17.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:17.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:43:17.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:43:17.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:43:17.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:43:17.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:43:17.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:17.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:17.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:43:17.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:43:17.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:43:18.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:43:18.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:43:18.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:43:18.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:43:18.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:43:18.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:43:18.214 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:43:18.214 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:43:18.214 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:43:18.214 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:18.215 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:18.215 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:18.215 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:43:18.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:43:18.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:18.289 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:18.290 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:18.290 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:43:18.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:43:18.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:43:18.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:43:18.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:43:18.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:43:18.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:43:18.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:43:18.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:43:18.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:43:19.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:43:19.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:43:19.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:43:19.191 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:43:19.191 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:19.191 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:19.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:43:19.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:43:19.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:43:19.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:43:19.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:43:19.466 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:43:19.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:43:20.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:43:20.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:43:20.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:43:20.477 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:43:20.477 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:20.477 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:20.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:43:20.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:43:20.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:43:20.544 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:43:20.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:43:20.553 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:43:18.005+0000 7f185521c900 0 load: jerasure load: lrc 2024-09-15T02:43:20.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-09-15T02:43:20.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-09-15T02:43:20.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-09-15T02:43:20.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:43:20.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:43:20.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-09-15T02:43:20.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T02:43:20.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:43:20.555 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T02:43:20.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T02:43:20.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:43:20.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:43:20.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:43:20.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:43:20.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:43:20.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:43:20.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:43:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:43:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:43:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:43:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:43:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:43:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:43:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:43:20.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d391d7e-c57a-4d2e-b4f5-c580d8423897 2024-09-15T02:43:20.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8d391d7e-c57a-4d2e-b4f5-c580d8423897' 2024-09-15T02:43:20.564 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 8d391d7e-c57a-4d2e-b4f5-c580d8423897 2024-09-15T02:43:20.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:43:20.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDISeZmMtdWIhAACPMT7upCxWAnr9d3HPuexA== 2024-09-15T02:43:20.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDISeZmMtdWIhAACPMT7upCxWAnr9d3HPuexA=="}' 2024-09-15T02:43:20.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d391d7e-c57a-4d2e-b4f5-c580d8423897 -i td/test-erasure-eio/0/new.json 2024-09-15T02:43:20.852 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:43:20.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:43:20.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDISeZmMtdWIhAACPMT7upCxWAnr9d3HPuexA== --osd-uuid 8d391d7e-c57a-4d2e-b4f5-c580d8423897 2024-09-15T02:43:20.892 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:20.891+0000 7feda111e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:20.898 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:20.898+0000 7feda111e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:20.899 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:20.899+0000 7feda111e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:20.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:20.899+0000 7feda111e440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:43:20.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:20.899+0000 7feda111e440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:43:20.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:20.899+0000 7feda111e440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:43:20.931 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:20.930+0000 7feda111e440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:43:26.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:43:26.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:43:26.709 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:43:26.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:43:26.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:43:27.194 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:43:27.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:43:27.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:43:27.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:43:27.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:43:27.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:43:27.221 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:27.220+0000 7f6c91636440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:27.223 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:27.222+0000 7f6c91636440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:27.224 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:27.223+0000 7f6c91636440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:43:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:43:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:43:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:43:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:43:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:43:27.587 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:43:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:27.742 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:27.741+0000 7f6c91636440 -1 Falling back to public interface 2024-09-15T02:43:27.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:28.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:28.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:28.938 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:43:28.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:43:28.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:28.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:29.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:30.216 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:30.215+0000 7f6c91636440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:43:30.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:30.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:30.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:43:30.296 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:43:30.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:30.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:30.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:31.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:31.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:31.821 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:43:31.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:43:31.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:31.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:32.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:33.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:33.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:33.207 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:43:33.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:43:33.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:33.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:33.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:34.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:34.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:34.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:43:34.598 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:43:34.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:34.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:35.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:36.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:36.094 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:43:36.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:36.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:43:36.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:36.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:36.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:37.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:37.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:37.508 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:43:37.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:43:37.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:37.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:38.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:39.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:39.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:39.007 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:43:39.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:43:39.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:39.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:39.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:40.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:40.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:40.373 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:43:40.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:43:40.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:40.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:40.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:41.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:41.815 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:43:41.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:41.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:43:41.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:41.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:42.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:43.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:43.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:43.638 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:43:43.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:43:43.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:43.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:44.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:45.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:45.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:45.479 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:43:45.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:43:45.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:45.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:46.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:47.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:47.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:47.386 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:43:47.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:43:47.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:47.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:48.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:49.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:49.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:49.174 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:43:49.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:43:49.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:49.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:50.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:51.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:51.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:51.040 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:43:51.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:43:51.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:51.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:51.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:43:52.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:43:52.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:43:52.733 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:43:52.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:43:52.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:43:52.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:43:53.121 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/159782,v1:127.0.0.1:6803/159782] [v2:127.0.0.1:6804/159782,v1:127.0.0.1:6805/159782] exists,up 8d391d7e-c57a-4d2e-b4f5-c580d8423897 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:43:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:43:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:43:53.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0fc69b59-5a8e-4bd5-8d99-29160fbb8a2a 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0fc69b59-5a8e-4bd5-8d99-29160fbb8a2a' 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 0fc69b59-5a8e-4bd5-8d99-29160fbb8a2a 2024-09-15T02:43:53.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:43:53.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpSeZmsUFwCBAAN7j3AWigbi03nbyOiX2GZg== 2024-09-15T02:43:53.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpSeZmsUFwCBAAN7j3AWigbi03nbyOiX2GZg=="}' 2024-09-15T02:43:53.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0fc69b59-5a8e-4bd5-8d99-29160fbb8a2a -i td/test-erasure-eio/1/new.json 2024-09-15T02:43:53.608 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:43:53.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:43:53.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpSeZmsUFwCBAAN7j3AWigbi03nbyOiX2GZg== --osd-uuid 0fc69b59-5a8e-4bd5-8d99-29160fbb8a2a 2024-09-15T02:43:53.648 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:53.646+0000 7efd47321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:53.650 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:53.648+0000 7efd47321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:53.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:53.649+0000 7efd47321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:53.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:53.649+0000 7efd47321440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:43:53.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:53.649+0000 7efd47321440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:43:53.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:53.649+0000 7efd47321440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:43:53.690 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:53.689+0000 7efd47321440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:43:59.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:43:59.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:43:59.237 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:43:59.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:43:59.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:43:59.755 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:43:59.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:43:59.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:43:59.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:43:59.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:43:59.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:43:59.782 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:59.781+0000 7fe49d11e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:59.784 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:59.784+0000 7fe49d11e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:43:59.786 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:43:59.785+0000 7fe49d11e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:00.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:44:00.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:44:00.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:44:00.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:44:00.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:44:00.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:00.112 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:44:00.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:44:00.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:00.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:00.308 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:00.307+0000 7fe49d11e440 -1 Falling back to public interface 2024-09-15T02:44:00.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:01.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:01.468 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:44:01.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:44:01.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:01.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:01.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:02.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:02.800+0000 7fe49d11e440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:44:02.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:02.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:02.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:44:02.834 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:44:02.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:02.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:03.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:04.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:04.245 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:44:04.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:04.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:44:04.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:04.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:04.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:05.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:05.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:05.668 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:44:05.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:44:05.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:05.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:06.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:07.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:07.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:07.073 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:44:07.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:44:07.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:07.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:07.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:08.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:08.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:08.531 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:44:08.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:44:08.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:08.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:08.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:09.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:09.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:09.911 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:44:09.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:44:09.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:09.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:10.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:11.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:11.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:11.277 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:44:11.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:44:11.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:11.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:11.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:12.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:12.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:12.642 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:44:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:44:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:13.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:14.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:14.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:14.049 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:44:14.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:44:14.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:14.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:14.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:15.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:15.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:15.980 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:44:15.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:44:15.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:15.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:16.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:17.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:17.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:17.536 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:44:17.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:44:17.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:17.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:18.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:19.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:19.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:19.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:20.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:20.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:20.935 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:44:20.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:44:20.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:20.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:22.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:23.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:23.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:23.199 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:44:23.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:44:23.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:23.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:24.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:25.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:25.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:25.048 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:44:25.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:44:25.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:25.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:44:25.604 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/161511,v1:127.0.0.1:6811/161511] [v2:127.0.0.1:6812/161511,v1:127.0.0.1:6813/161511] exists,up 0fc69b59-5a8e-4bd5-8d99-29160fbb8a2a 2024-09-15T02:44:25.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:44:25.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:44:25.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:44:25.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:44:25.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d25cd96-daf3-4ec7-885c-76f453498ff7 2024-09-15T02:44:25.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9d25cd96-daf3-4ec7-885c-76f453498ff7' 2024-09-15T02:44:25.611 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 9d25cd96-daf3-4ec7-885c-76f453498ff7 2024-09-15T02:44:25.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:44:25.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJSuZmHvI9JRAArUehssrg7FnkFh6ejrvJqw== 2024-09-15T02:44:25.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJSuZmHvI9JRAArUehssrg7FnkFh6ejrvJqw=="}' 2024-09-15T02:44:25.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d25cd96-daf3-4ec7-885c-76f453498ff7 -i td/test-erasure-eio/2/new.json 2024-09-15T02:44:26.092 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:44:26.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:44:26.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJSuZmHvI9JRAArUehssrg7FnkFh6ejrvJqw== --osd-uuid 9d25cd96-daf3-4ec7-885c-76f453498ff7 2024-09-15T02:44:26.132 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:26.131+0000 7f0848966440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:26.133 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:26.133+0000 7f0848966440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:26.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:26.134+0000 7f0848966440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:26.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:26.134+0000 7f0848966440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:44:26.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:26.134+0000 7f0848966440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:44:26.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:26.134+0000 7f0848966440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:44:26.207 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:26.206+0000 7f0848966440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:44:32.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:44:32.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:44:32.242 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:44:32.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:44:32.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:44:32.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:44:32.730 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:44:32.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:44:32.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:44:32.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:44:32.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:44:32.757 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:32.756+0000 7f4d8f920440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:32.759 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:32.758+0000 7f4d8f920440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:32.760 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:32.759+0000 7f4d8f920440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:44:33.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:44:33.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:44:33.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:44:33.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:44:33.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:44:33.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:33.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:44:33.123 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:44:33.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:33.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:33.276 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:33.275+0000 7f4d8f920440 -1 Falling back to public interface 2024-09-15T02:44:33.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:34.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:34.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:34.521 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:44:34.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:44:34.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:34.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:34.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:35.872 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:44:35.870+0000 7f4d8f920440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:44:35.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:35.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:35.884 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:44:35.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:44:35.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:35.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:36.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:37.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:37.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:37.337 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:44:37.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:44:37.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:37.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:37.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:38.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:38.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:38.713 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:44:38.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:44:38.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:38.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:39.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:40.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:40.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:40.161 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:44:40.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:44:40.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:40.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:40.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:41.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:41.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:41.670 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:44:41.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:44:41.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:41.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:42.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:43.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:43.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:43.085 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:44:43.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:44:43.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:43.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:43.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:44.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:44.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:44.591 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:44:44.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:44:44.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:44.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:45.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:46.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:46.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:46.046 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:44:46.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:44:46.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:46.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:46.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:47.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:47.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:47.452 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:44:47.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:44:47.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:47.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:48.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:49.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:49.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:49.191 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:44:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:44:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:50.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:51.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:51.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:51.245 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:44:51.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:44:51.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:51.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:52.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:53.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:53.062 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:44:53.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:44:53.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:53.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:54.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:55.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:55.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:55.203 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:44:55.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:44:55.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:55.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:56.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:57.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:57.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:57.033 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:44:57.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:44:57.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:57.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:58.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:44:59.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:44:59.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:44:59.058 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:44:59.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:44:59.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:44:59.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:44:59.478 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/163240,v1:127.0.0.1:6819/163240] [v2:127.0.0.1:6820/163240,v1:127.0.0.1:6821/163240] exists,up 9d25cd96-daf3-4ec7-885c-76f453498ff7 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:44:59.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:44:59.483 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:44:59.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:44:59.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:44:59.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:44:59.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:44:59.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dbc229da-0013-42a6-8fb6-2386d3fbaf57 2024-09-15T02:44:59.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 dbc229da-0013-42a6-8fb6-2386d3fbaf57' 2024-09-15T02:44:59.485 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 dbc229da-0013-42a6-8fb6-2386d3fbaf57 2024-09-15T02:44:59.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:44:59.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArSuZmWYy0HRAAUonj5bG1k7Yj5cpFnWh0Aw== 2024-09-15T02:44:59.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArSuZmWYy0HRAAUonj5bG1k7Yj5cpFnWh0Aw=="}' 2024-09-15T02:44:59.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dbc229da-0013-42a6-8fb6-2386d3fbaf57 -i td/test-erasure-eio/3/new.json 2024-09-15T02:45:00.335 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:45:00.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:45:00.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArSuZmWYy0HRAAUonj5bG1k7Yj5cpFnWh0Aw== --osd-uuid dbc229da-0013-42a6-8fb6-2386d3fbaf57 2024-09-15T02:45:00.376 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:00.375+0000 7f5a40c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:00.444 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:00.443+0000 7f5a40c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:00.446 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:00.444+0000 7f5a40c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:00.446 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:00.445+0000 7f5a40c0f440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:45:00.446 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:00.445+0000 7f5a40c0f440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:45:00.446 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:00.445+0000 7f5a40c0f440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:45:00.509 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:00.508+0000 7f5a40c0f440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:45:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:45:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:45:06.209 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:45:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:45:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:45:06.695 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:45:06.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:45:06.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:45:06.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:45:06.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:45:06.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:45:06.722 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:06.721+0000 7fe53d030440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:06.724 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:06.723+0000 7fe53d030440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:06.725 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:06.724+0000 7fe53d030440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:07.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:45:07.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:45:07.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:45:07.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:45:07.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:45:07.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:07.098 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:45:07.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:45:07.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:07.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:07.239 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:07.238+0000 7fe53d030440 -1 Falling back to public interface 2024-09-15T02:45:07.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:08.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:08.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:08.460 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:45:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:45:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:08.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:09.807 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:09.805+0000 7fe53d030440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:45:09.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:09.819 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:45:09.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:09.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:45:09.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:09.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:10.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:11.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:11.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:11.249 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:45:11.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:45:11.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:11.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:11.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:12.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:12.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:12.908 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:45:12.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:45:12.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:12.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:13.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:14.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:14.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:14.290 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:45:14.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:45:14.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:14.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:14.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:15.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:15.691 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:45:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:45:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:15.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:16.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:17.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:17.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:17.104 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:45:17.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:45:17.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:17.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:17.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:18.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:18.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:18.553 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:45:18.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:45:18.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:18.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:18.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:19.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:19.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:19.915 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:45:19.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:45:19.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:19.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:20.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:21.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:21.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:21.323 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:45:21.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:45:21.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:21.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:21.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:22.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:22.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:22.697 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:45:22.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:45:22.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:22.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:23.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:24.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:24.465 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:45:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:45:24.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:24.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:25.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:26.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:26.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:26.145 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:45:26.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:45:26.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:26.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:26.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:27.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:27.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:27.848 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:45:27.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:45:27.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:27.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:28.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:29.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:29.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:29.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:45:29.597 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:45:29.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:29.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:30.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:31.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:31.422 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:45:31.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:31.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:45:31.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:31.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:32.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:33.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:33.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:33.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:45:33.058 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:45:33.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:33.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:45:33.444 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/164969,v1:127.0.0.1:6827/164969] [v2:127.0.0.1:6828/164969,v1:127.0.0.1:6829/164969] exists,up dbc229da-0013-42a6-8fb6-2386d3fbaf57 2024-09-15T02:45:33.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:45:33.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:45:33.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:45:33.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T02:45:33.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:45:33.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd457da6-144c-4b4f-a064-af6ca88448c9 2024-09-15T02:45:33.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 cd457da6-144c-4b4f-a064-af6ca88448c9' 2024-09-15T02:45:33.453 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 cd457da6-144c-4b4f-a064-af6ca88448c9 2024-09-15T02:45:33.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:45:33.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNSuZmnsatGxAAA6j+GP/9Iiw9LlJxHX9+Fw== 2024-09-15T02:45:33.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNSuZmnsatGxAAA6j+GP/9Iiw9LlJxHX9+Fw=="}' 2024-09-15T02:45:33.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd457da6-144c-4b4f-a064-af6ca88448c9 -i td/test-erasure-eio/4/new.json 2024-09-15T02:45:34.052 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:45:34.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T02:45:34.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNSuZmnsatGxAAA6j+GP/9Iiw9LlJxHX9+Fw== --osd-uuid cd457da6-144c-4b4f-a064-af6ca88448c9 2024-09-15T02:45:34.093 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:34.091+0000 7ff6d8f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:34.247 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:34.245+0000 7ff6d8f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:34.247 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:34.246+0000 7ff6d8f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:34.248 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:34.247+0000 7ff6d8f21440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:45:34.248 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:34.247+0000 7ff6d8f21440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:45:34.248 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:34.247+0000 7ff6d8f21440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:45:34.321 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:34.320+0000 7ff6d8f21440 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T02:45:41.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T02:45:41.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:45:41.701 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:45:41.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:45:41.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:45:42.189 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:45:42.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:45:42.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:45:42.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:45:42.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:45:42.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:45:42.217 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:42.214+0000 7fbb97127440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:42.218 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:42.217+0000 7fbb97127440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:42.219 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:42.218+0000 7fbb97127440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:45:42.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:45:42.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:45:42.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:45:42.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:45:42.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:45:42.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:42.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:45:42.542 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:45:42.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:42.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:42.736 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:42.734+0000 7fbb97127440 -1 Falling back to public interface 2024-09-15T02:45:42.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:43.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:43.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:43.895 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:45:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:45:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:44.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:45.253 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:45:45.251+0000 7fbb97127440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:45:45.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:45.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:45.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:45:45.257 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:45:45.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:45.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:45.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:46.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:46.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:46.618 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:45:46.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:45:46.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:46.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:47.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:48.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:48.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:48.272 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:45:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:45:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:48.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:48.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:49.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:49.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:49.663 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:45:49.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:45:49.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:49.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:50.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:51.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:51.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:51.072 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:45:51.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:45:51.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:51.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:51.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:52.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:52.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:52.494 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:45:52.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:45:52.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:52.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:52.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:53.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:53.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:53.861 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:45:53.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:45:53.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:53.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:54.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:55.289 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:45:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:45:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:56.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:56.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:56.731 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:45:56.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:45:56.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:56.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:57.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:58.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:58.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:58.236 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:45:58.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:45:58.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:58.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:45:58.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:45:59.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:45:59.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:45:59.691 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:45:59.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:45:59.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:45:59.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:46:00.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:01.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:01.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:01.660 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:46:01.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:46:01.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:01.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:46:02.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:03.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:03.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:03.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:46:03.472 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:46:03.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:03.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:46:04.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:05.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:05.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:05.241 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:46:05.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:46:05.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:05.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:46:06.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:07.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:07.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:07.110 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:46:07.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:46:07.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:07.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:46:07.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:08.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:08.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:08.930 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:46:08.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:46:08.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:08.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:46:09.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:10.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:10.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:10.386 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:46:10.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:46:10.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:10.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:46:10.966 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/166723,v1:127.0.0.1:6835/166723] [v2:127.0.0.1:6836/166723,v1:127.0.0.1:6837/166723] exists,up cd457da6-144c-4b4f-a064-af6ca88448c9 2024-09-15T02:46:10.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:46:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:46:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:46:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:46:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T02:46:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:46:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:46:10.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T02:46:10.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:46:10.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T02:46:10.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:46:10.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:46:10.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:46:10.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:46:10.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:46:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:46:10.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46447286-543d-447b-8076-2f09c94a3fee 2024-09-15T02:46:10.975 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 46447286-543d-447b-8076-2f09c94a3fee 2024-09-15T02:46:10.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 46447286-543d-447b-8076-2f09c94a3fee' 2024-09-15T02:46:10.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:46:10.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBySuZmqYzJOhAAXp5sM8GkbbCa11zivvOVlQ== 2024-09-15T02:46:10.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBySuZmqYzJOhAAXp5sM8GkbbCa11zivvOVlQ=="}' 2024-09-15T02:46:10.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46447286-543d-447b-8076-2f09c94a3fee -i td/test-erasure-eio/5/new.json 2024-09-15T02:46:12.037 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:46:12.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T02:46:12.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBySuZmqYzJOhAAXp5sM8GkbbCa11zivvOVlQ== --osd-uuid 46447286-543d-447b-8076-2f09c94a3fee 2024-09-15T02:46:12.078 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:12.077+0000 7f715a00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:12.106 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:12.105+0000 7f715a00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:12.106+0000 7f715a00d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:12.106+0000 7f715a00d440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:46:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:12.106+0000 7f715a00d440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:46:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:12.106+0000 7f715a00d440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:46:12.246 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:12.245+0000 7f715a00d440 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T02:46:18.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T02:46:18.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:46:18.642 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-15T02:46:18.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T02:46:18.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:46:19.132 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-15T02:46:19.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T02:46:19.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:46:19.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:46:19.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:46:19.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:46:19.159 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:19.158+0000 7fa17ac0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:19.184 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:19.183+0000 7fa17ac0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:19.185 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:19.184+0000 7fa17ac0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T02:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T02:46:19.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:46:19.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:46:19.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:19.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:46:19.557 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:46:19.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:19.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:19.713 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:19.712+0000 7fa17ac0b440 -1 Falling back to public interface 2024-09-15T02:46:19.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:20.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:20.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:20.912 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:46:20.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:46:20.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:20.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:21.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:22.197 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:22.196+0000 7fa17ac0b440 -1 osd.5 0 log_to_monitors true 2024-09-15T02:46:22.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:22.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:22.278 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:46:22.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:46:22.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:22.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:22.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:23.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:23.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:23.921 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:46:23.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:46:23.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:23.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:24.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:25.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:25.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:25.408 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:46:25.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:46:25.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:25.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:25.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:26.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:26.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:26.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:46:26.876 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:46:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:28.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:28.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:28.340 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:46:28.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:46:28.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:28.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:28.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:29.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:29.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:29.700 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:46:29.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:46:29.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:29.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:30.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:31.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:31.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:31.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:46:31.143 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:46:31.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:31.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:31.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:32.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:32.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:32.577 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:46:32.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:46:32.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:32.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:33.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:34.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:34.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:34.074 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:46:34.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:46:34.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:34.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:34.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:35.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:35.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:35.472 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:46:35.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:46:35.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:35.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:36.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:37.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:37.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:37.188 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:46:37.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:46:37.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:37.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:38.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:39.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:39.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:39.022 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:46:39.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:46:39.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:39.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:40.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:41.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:41.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:41.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:46:41.138 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:46:41.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:41.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:41.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:42.987 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:46:42.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:46:42.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:42.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:43.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:44.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:44.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:44.848 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:46:44.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:46:44.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:44.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:47.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:47.032 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:46:47.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:47.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:46:47.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:47.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:46:47.478 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/168502,v1:127.0.0.1:6843/168502] [v2:127.0.0.1:6844/168502,v1:127.0.0.1:6845/168502] exists,up 46447286-543d-447b-8076-2f09c94a3fee 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:46:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:46:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:46:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T02:46:47.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:46:47.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db43a4e6-77cb-47e3-b551-896492cbc222 2024-09-15T02:46:47.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 db43a4e6-77cb-47e3-b551-896492cbc222' 2024-09-15T02:46:47.486 INFO:tasks.workunit.client.0.smithi130.stdout:add osd6 db43a4e6-77cb-47e3-b551-896492cbc222 2024-09-15T02:46:47.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:46:47.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXSuZmTs27HRAAIgu6CbJQcQpIsShss6IzEQ== 2024-09-15T02:46:47.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXSuZmTs27HRAAIgu6CbJQcQpIsShss6IzEQ=="}' 2024-09-15T02:46:47.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db43a4e6-77cb-47e3-b551-896492cbc222 -i td/test-erasure-eio/6/new.json 2024-09-15T02:46:48.113 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:46:48.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T02:46:48.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXSuZmTs27HRAAIgu6CbJQcQpIsShss6IzEQ== --osd-uuid db43a4e6-77cb-47e3-b551-896492cbc222 2024-09-15T02:46:48.153 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:48.152+0000 7fdf4531f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:48.235+0000 7fdf4531f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:48.236+0000 7fdf4531f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:48.237 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:48.236+0000 7fdf4531f440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:46:48.237 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:48.236+0000 7fdf4531f440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:46:48.237 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:48.236+0000 7fdf4531f440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:46:48.322 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:48.321+0000 7fdf4531f440 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T02:46:54.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T02:46:54.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:46:54.438 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd6 key to auth repository 2024-09-15T02:46:54.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-15T02:46:54.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:46:54.933 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.6 2024-09-15T02:46:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-15T02:46:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:46:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:46:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:46:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:46:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:54.959+0000 7f4e0b20c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:54.972 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:54.971+0000 7f4e0b20c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:54.973 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:54.973+0000 7f4e0b20c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:46:55.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-15T02:46:55.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:46:55.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-15T02:46:55.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:46:55.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:46:55.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:55.308 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:46:55.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:46:55.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:55.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:46:55.505 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:55.504+0000 7f4e0b20c440 -1 Falling back to public interface 2024-09-15T02:46:55.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:56.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:56.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:56.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:46:56.730 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:46:56.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:56.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:46:57.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:58.036 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:46:58.035+0000 7f4e0b20c440 -1 osd.6 0 log_to_monitors true 2024-09-15T02:46:58.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:58.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:58.106 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:46:58.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:46:58.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:58.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:46:58.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:46:59.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:46:59.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:46:59.563 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:46:59.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:46:59.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:46:59.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:00.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:01.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:01.146 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:47:01.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:01.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:47:01.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:01.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:01.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:02.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:02.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:02.510 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:47:02.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:47:02.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:02.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:02.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:03.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:03.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:03.976 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:47:03.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:47:03.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:03.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:04.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:05.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:05.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:05.348 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:47:05.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:47:05.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:05.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:05.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:06.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:06.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:06.883 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:47:06.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:47:06.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:06.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:07.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:08.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:08.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:08.278 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:47:08.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:47:08.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:08.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:08.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:09.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:09.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:09.693 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:47:09.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:47:09.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:09.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:10.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:11.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:11.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:11.141 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:47:11.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:47:11.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:11.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:12.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:13.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:13.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:13.274 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:47:13.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:47:13.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:13.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:14.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:15.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:15.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:15.132 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:47:15.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:47:15.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:15.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:16.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:17.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:17.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:17.019 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:47:17.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:47:17.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:17.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:17.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:18.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:18.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:18.811 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:47:18.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:47:18.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:18.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:19.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:20.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:20.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:20.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:47:20.625 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:47:20.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:20.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:21.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:22.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:22.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:22.454 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:47:22.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:47:22.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:22.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:47:22.935 INFO:tasks.workunit.client.0.smithi130.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/170256,v1:127.0.0.1:6851/170256] [v2:127.0.0.1:6852/170256,v1:127.0.0.1:6853/170256] exists,up db43a4e6-77cb-47e3-b551-896492cbc222 2024-09-15T02:47:22.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:47:22.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:47:23.012 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:47:23.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:47:23.023 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:43:27.745+0000 7f6c91636440 0 load: jerasure load: lrc 2024-09-15T02:47:23.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:47:23.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-09-15T02:47:23.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T02:47:23.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:47:23.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:47:23.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T02:47:23.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:47:23.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T02:47:23.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:47:23.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T02:47:23.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:47:23.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:47:25.214 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:47:25.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:47:26.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:47:26.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:47:26.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:47:26.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:47:26.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:47:26.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:47:26.228 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:47:26.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:47:26.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:47:26.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:47:26.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:47:26.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:47:26.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:47:26.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:47:26.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:47:26.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:47:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:47:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:47:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:47:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:47:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:47:26.675 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:47:26.675 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:47:26.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:47:26.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:26.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:47:26.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803824 2024-09-15T02:47:26.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803824 2024-09-15T02:47:26.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824' 2024-09-15T02:47:26.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:26.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:47:26.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574891 2024-09-15T02:47:26.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574891 2024-09-15T02:47:26.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891' 2024-09-15T02:47:26.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:26.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:47:27.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=77309411364 2024-09-15T02:47:27.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 77309411364 2024-09-15T02:47:27.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-77309411364' 2024-09-15T02:47:27.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:27.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:47:27.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084318 2024-09-15T02:47:27.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084318 2024-09-15T02:47:27.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-77309411364 3-120259084318' 2024-09-15T02:47:27.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:27.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:47:27.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691863 2024-09-15T02:47:27.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691863 2024-09-15T02:47:27.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-77309411364 3-120259084318 4-171798691863' 2024-09-15T02:47:27.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:27.373 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:47:27.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332107 2024-09-15T02:47:27.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332107 2024-09-15T02:47:27.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-77309411364 3-120259084318 4-171798691863 5-219043332107' 2024-09-15T02:47:27.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:27.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:47:27.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070470 2024-09-15T02:47:27.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070470 2024-09-15T02:47:27.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-77309411364 3-120259084318 4-171798691863 5-219043332107 6-253403070470' 2024-09-15T02:47:27.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:27.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803824 2024-09-15T02:47:27.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:27.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:47:27.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803824 2024-09-15T02:47:27.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:27.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803824 2024-09-15T02:47:27.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-09-15T02:47:27.586 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803824 2024-09-15T02:47:27.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:47:27.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803822 -lt 25769803824 2024-09-15T02:47:27.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:47:28.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:47:28.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:47:29.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803825 -lt 25769803824 2024-09-15T02:47:29.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:29.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574891 2024-09-15T02:47:29.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:29.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:47:29.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574891 2024-09-15T02:47:29.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:29.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574891 2024-09-15T02:47:29.293 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574891 2024-09-15T02:47:29.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574891' 2024-09-15T02:47:29.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:47:29.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574891 -lt 55834574891 2024-09-15T02:47:29.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:29.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-77309411364 2024-09-15T02:47:29.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:29.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:47:29.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-77309411364 2024-09-15T02:47:29.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:29.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=77309411364 2024-09-15T02:47:29.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 77309411364' 2024-09-15T02:47:29.645 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 77309411364 2024-09-15T02:47:29.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:47:29.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 77309411365 -lt 77309411364 2024-09-15T02:47:29.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:29.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084318 2024-09-15T02:47:29.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:29.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:47:29.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084318 2024-09-15T02:47:29.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:29.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084318 2024-09-15T02:47:29.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084318' 2024-09-15T02:47:29.999 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084318 2024-09-15T02:47:29.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:47:30.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084319 -lt 120259084318 2024-09-15T02:47:30.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:30.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691863 2024-09-15T02:47:30.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:30.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:47:30.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691863 2024-09-15T02:47:30.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:30.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691863 2024-09-15T02:47:30.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691863' 2024-09-15T02:47:30.356 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 171798691863 2024-09-15T02:47:30.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:47:30.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691863 -lt 171798691863 2024-09-15T02:47:30.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:30.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-219043332107 2024-09-15T02:47:30.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:30.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:47:30.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-219043332107 2024-09-15T02:47:30.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:30.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332107 2024-09-15T02:47:30.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 219043332107' 2024-09-15T02:47:30.705 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 219043332107 2024-09-15T02:47:30.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:47:31.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332107 -lt 219043332107 2024-09-15T02:47:31.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:31.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-253403070470 2024-09-15T02:47:31.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:31.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:47:31.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-253403070470 2024-09-15T02:47:31.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070470 2024-09-15T02:47:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 253403070470' 2024-09-15T02:47:31.065 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 253403070470 2024-09-15T02:47:31.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:47:31.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070470 -lt 253403070470 2024-09-15T02:47:31.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:47:31.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:47:31.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:47:31.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:47:32.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:47:32.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:47:32.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:47:32.251 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:47:32.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:47:32.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:47:32.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:47:32.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:47:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-15T02:47:32.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:47:32.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-15T02:47:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-15T02:47:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:47:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:47:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-15T02:47:33.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:47:33.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-15T02:47:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-15T02:47:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:47:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:47:33.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:47:33.540 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:47:33.541 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-15T02:47:33.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:47:33.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T02:47:33.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:47:33.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T02:47:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:47:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T02:47:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:47:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T02:47:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:47:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:47:33.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:47:33.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:47:33.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:47:33.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-15T02:47:34.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-09-15T02:47:34.544 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2024-09-15T02:47:34.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-09-15T02:47:34.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:47:34.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:47:34.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:47:34.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:47:34.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:47:34.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:47:34.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:47:34.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:47:34.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:47:34.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:47:34.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:47:34.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:47:34.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:47:34.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:47:34.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:47:35.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:47:35.162 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:47:35.162 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:47:35.162 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:47:35.162 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:47:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:47:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:47:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:47:35.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:35.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:47:35.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803828 2024-09-15T02:47:35.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803828 2024-09-15T02:47:35.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828' 2024-09-15T02:47:35.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:35.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:47:35.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574895 2024-09-15T02:47:35.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574895 2024-09-15T02:47:35.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-55834574895' 2024-09-15T02:47:35.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:35.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:47:35.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=77309411368 2024-09-15T02:47:35.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 77309411368 2024-09-15T02:47:35.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-55834574895 2-77309411368' 2024-09-15T02:47:35.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:35.621 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:47:35.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084322 2024-09-15T02:47:35.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084322 2024-09-15T02:47:35.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-55834574895 2-77309411368 3-120259084322' 2024-09-15T02:47:35.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:35.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:47:36.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691867 2024-09-15T02:47:36.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691867 2024-09-15T02:47:36.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-55834574895 2-77309411368 3-120259084322 4-171798691867' 2024-09-15T02:47:36.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:36.006 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:47:36.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332110 2024-09-15T02:47:36.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332110 2024-09-15T02:47:36.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-55834574895 2-77309411368 3-120259084322 4-171798691867 5-219043332110' 2024-09-15T02:47:36.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:47:36.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:47:36.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070473 2024-09-15T02:47:36.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070473 2024-09-15T02:47:36.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-55834574895 2-77309411368 3-120259084322 4-171798691867 5-219043332110 6-253403070473' 2024-09-15T02:47:36.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:36.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803828 2024-09-15T02:47:36.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:36.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:47:36.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803828 2024-09-15T02:47:36.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:36.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803828 2024-09-15T02:47:36.364 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803828 2024-09-15T02:47:36.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-09-15T02:47:36.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:47:36.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 0 -lt 25769803828 2024-09-15T02:47:36.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:47:37.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:47:37.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:47:38.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803828 -lt 25769803828 2024-09-15T02:47:38.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:38.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574895 2024-09-15T02:47:38.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:38.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:47:38.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574895 2024-09-15T02:47:38.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:38.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574895 2024-09-15T02:47:38.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-09-15T02:47:38.073 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574895 2024-09-15T02:47:38.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:47:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574895 -lt 55834574895 2024-09-15T02:47:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:38.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-77309411368 2024-09-15T02:47:38.463 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:38.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:47:38.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-77309411368 2024-09-15T02:47:38.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:38.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=77309411368 2024-09-15T02:47:38.466 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 77309411368 2024-09-15T02:47:38.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 77309411368' 2024-09-15T02:47:38.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:47:38.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 77309411368 -lt 77309411368 2024-09-15T02:47:38.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:38.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084322 2024-09-15T02:47:38.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:38.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:47:38.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084322 2024-09-15T02:47:38.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:38.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084322 2024-09-15T02:47:38.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084322' 2024-09-15T02:47:38.823 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084322 2024-09-15T02:47:38.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:47:39.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084323 -lt 120259084322 2024-09-15T02:47:39.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:39.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691867 2024-09-15T02:47:39.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:39.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:47:39.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691867 2024-09-15T02:47:39.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:39.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691867 2024-09-15T02:47:39.170 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 171798691867 2024-09-15T02:47:39.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691867' 2024-09-15T02:47:39.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:47:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691867 -lt 171798691867 2024-09-15T02:47:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:39.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-219043332110 2024-09-15T02:47:39.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:39.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:47:39.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-219043332110 2024-09-15T02:47:39.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:39.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332110 2024-09-15T02:47:39.519 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 219043332110 2024-09-15T02:47:39.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 219043332110' 2024-09-15T02:47:39.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:47:39.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332111 -lt 219043332110 2024-09-15T02:47:39.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:47:39.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-253403070473 2024-09-15T02:47:39.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:47:39.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:47:39.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-253403070473 2024-09-15T02:47:39.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:47:39.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070473 2024-09-15T02:47:39.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 253403070473' 2024-09-15T02:47:39.869 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 253403070473 2024-09-15T02:47:39.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:47:40.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070474 -lt 253403070473 2024-09-15T02:47:40.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:47:40.215 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:47:40.216 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:47:40.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:47:41.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:47:41.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:47:41.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:47:41.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:47:41.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-15T02:47:41.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:47:41.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:47:41.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-15T02:47:41.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:47:41.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:47:41.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-15T02:47:41.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:47:41.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:47:41.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-15T02:47:41.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:47:41.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:47:41.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-15T02:47:41.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:47:41.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:47:41.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-15T02:47:41.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:47:41.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:47:41.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-15T02:47:41.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:47:41.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:47:41.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-09-15T02:47:41.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:47:41.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:47:42.191 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:47:42.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T02:47:42.817 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T02:47:42.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T02:47:42.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:47:42.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:47:42.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:47:42.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:47:42.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:47:42.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:47:42.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:47:42.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:47:42.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:47:42.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:47:42.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:47:42.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:47:42.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:47:42.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:47:42.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:47:42.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:42.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:47:42.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:47:42.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:42.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:47:42.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:47:42.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:47:42.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:47:42.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:42.978 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:42.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:47:42.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:47:42.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:47:42.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:47:42.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:47:42.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:47:42.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:47:42.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:47:42.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:47:42.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:47:42.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:47:42.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:47:42.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:42.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:47:42.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:47:42.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:47:42.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:47:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:47:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:47:42.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:47:42.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:42.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:42.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:47:42.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:47:42.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:47:42.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:47:42.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:47:42.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:42.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:42.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:47:42.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:47:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:47:42.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:47:43.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:47:43.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:47:43.332 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:47:43.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:47:43.332 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:43.333 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:43.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:47:43.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:47:43.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:47:43.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:47:43.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:43.576 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:47:43.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:47:43.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:47:43.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:47:43.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:47:43.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:47:43.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:47:43.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:47:43.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:47:43.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:47:43.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:47:43.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:47:43.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:47:43.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:47:43.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:47:43.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:47:44.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:47:44.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:47:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:47:44.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:47:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:47:44.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:47:45.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:47:45.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:47:45.914 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:47:45.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:47:45.923 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:47:43.360+0000 7f4d954c3900 0 load: jerasure load: lrc 2024-09-15T02:47:45.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-09-15T02:47:45.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-09-15T02:47:45.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2024-09-15T02:47:45.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2024-09-15T02:47:45.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2024-09-15T02:47:45.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:47:45.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-15T02:47:45.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:47:45.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2024-09-15T02:47:45.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-15T02:47:45.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:47:45.925 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:47:45.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:47:45.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:47:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:47:45.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:47:45.932 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 b182ec24-9ecb-427d-88e5-21d17a8b042e 2024-09-15T02:47:45.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b182ec24-9ecb-427d-88e5-21d17a8b042e 2024-09-15T02:47:45.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b182ec24-9ecb-427d-88e5-21d17a8b042e' 2024-09-15T02:47:45.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:47:45.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRSuZmtt6FOBAA0q4rcm8bGu/5A+FLL1XqTQ== 2024-09-15T02:47:45.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRSuZmtt6FOBAA0q4rcm8bGu/5A+FLL1XqTQ=="}' 2024-09-15T02:47:45.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b182ec24-9ecb-427d-88e5-21d17a8b042e -i td/test-erasure-eio/0/new.json 2024-09-15T02:47:46.230 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:47:46.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:47:46.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRSuZmtt6FOBAA0q4rcm8bGu/5A+FLL1XqTQ== --osd-uuid b182ec24-9ecb-427d-88e5-21d17a8b042e 2024-09-15T02:47:46.273 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:46.272+0000 7f101fb57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:46.276 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:46.276+0000 7f101fb57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:46.277 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:46.277+0000 7f101fb57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:46.278 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:46.277+0000 7f101fb57440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:47:46.278 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:46.277+0000 7f101fb57440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:47:46.278 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:46.277+0000 7f101fb57440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:47:46.317 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:46.316+0000 7f101fb57440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:47:52.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:47:52.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:47:52.389 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:47:52.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:47:52.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:47:52.879 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:47:52.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:47:52.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:47:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:47:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:47:52.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:47:52.909 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:52.908+0000 7fef9c20a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:52.911 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:52.910+0000 7fef9c20a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:52.912 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:52.911+0000 7fef9c20a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:47:53.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:47:53.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:47:53.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:47:53.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:47:53.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:47:53.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:53.270 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:47:53.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:47:53.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:53.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:53.429 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:53.428+0000 7fef9c20a440 -1 Falling back to public interface 2024-09-15T02:47:53.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:54.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:54.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:54.628 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:47:54.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:47:54.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:54.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:54.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:55.900 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:47:55.899+0000 7fef9c20a440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:47:55.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:55.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:55.995 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:47:55.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:47:55.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:55.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:56.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:57.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:57.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:57.586 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:47:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:47:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:57.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:47:58.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:47:58.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:47:58.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:47:58.965 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:47:58.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:47:58.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:47:59.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:00.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:00.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:00.354 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:48:00.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:48:00.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:00.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:00.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:01.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:01.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:01.851 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:48:01.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:48:01.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:01.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:02.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:03.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:03.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:03.218 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:48:03.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:48:03.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:03.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:03.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:04.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:04.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:04.687 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:48:04.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:48:04.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:04.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:05.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:06.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:06.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:06.067 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:48:06.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:48:06.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:06.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:06.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:07.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:07.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:07.583 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:48:07.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:48:07.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:07.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:08.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:09.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:09.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:09.299 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:48:09.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:48:09.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:09.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:10.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:11.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:11.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:11.078 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:48:11.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:48:11.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:11.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:11.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:12.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:12.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:12.838 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:48:12.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:48:12.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:12.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:13.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:14.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:14.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:14.665 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:48:14.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:48:14.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:14.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:15.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:16.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:16.480 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:48:16.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:48:16.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:16.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:17.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:18.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:18.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:18.564 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:48:18.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:48:18.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:18.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:48:19.016 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/173703,v1:127.0.0.1:6803/173703] [v2:127.0.0.1:6804/173703,v1:127.0.0.1:6805/173703] exists,up b182ec24-9ecb-427d-88e5-21d17a8b042e 2024-09-15T02:48:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:48:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:48:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:48:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:48:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:48:19.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:48:19.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:48:19.023 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:48:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:48:19.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:48:19.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:48:19.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:48:19.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:48:19.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:48:19.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=96c6c3f0-782b-46ab-8ada-928c804c20c6 2024-09-15T02:48:19.025 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 96c6c3f0-782b-46ab-8ada-928c804c20c6 2024-09-15T02:48:19.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 96c6c3f0-782b-46ab-8ada-928c804c20c6' 2024-09-15T02:48:19.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:48:19.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzSuZmD5RWAhAAEoowtTmQ+c4pLv4MozNjBA== 2024-09-15T02:48:19.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzSuZmD5RWAhAAEoowtTmQ+c4pLv4MozNjBA=="}' 2024-09-15T02:48:19.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 96c6c3f0-782b-46ab-8ada-928c804c20c6 -i td/test-erasure-eio/1/new.json 2024-09-15T02:48:19.542 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:48:19.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:48:19.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzSuZmD5RWAhAAEoowtTmQ+c4pLv4MozNjBA== --osd-uuid 96c6c3f0-782b-46ab-8ada-928c804c20c6 2024-09-15T02:48:19.585 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:19.584+0000 7f7a46209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:19.588 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:19.586+0000 7f7a46209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:19.589 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:19.587+0000 7f7a46209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:19.589 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:19.587+0000 7f7a46209440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:48:19.589 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:19.587+0000 7f7a46209440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:48:19.589 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:19.587+0000 7f7a46209440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:48:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:19.665+0000 7f7a46209440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:48:25.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:48:25.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:48:25.786 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:48:25.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:48:25.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:48:26.288 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:48:26.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:48:26.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:48:26.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:48:26.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:48:26.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:48:26.319 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:26.317+0000 7f43cf50f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:26.336 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:26.334+0000 7f43cf50f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:26.337 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:26.335+0000 7f43cf50f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:26.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:48:26.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:48:26.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:48:26.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:48:26.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:48:26.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:26.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:48:26.684 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:48:26.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:26.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:26.868 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:26.866+0000 7f43cf50f440 -1 Falling back to public interface 2024-09-15T02:48:27.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:28.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:28.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:28.039 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:48:28.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:48:28.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:28.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:29.349 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:29.347+0000 7f43cf50f440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:48:29.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:29.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:29.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:48:29.404 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:48:29.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:29.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:30.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:31.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:31.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:31.035 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:48:31.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:48:31.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:31.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:31.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:32.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:32.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:32.570 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:48:32.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:48:32.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:32.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:33.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:34.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:34.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:34.029 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:48:34.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:48:34.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:34.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:34.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:35.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:35.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:35.413 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:48:35.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:48:35.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:35.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:35.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:36.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:36.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:36.786 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:48:36.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:48:36.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:36.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:37.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:38.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:38.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:38.289 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:48:38.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:48:38.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:38.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:38.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:39.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:39.693 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:48:39.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:48:39.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:39.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:40.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:41.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:41.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:41.150 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:48:41.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:48:41.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:41.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:41.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:42.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:42.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:42.541 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:48:42.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:48:42.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:42.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:43.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:44.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:44.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:44.385 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:48:44.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:48:44.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:44.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:45.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:46.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:46.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:46.393 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:48:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:48:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:46.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:47.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:48.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:48.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:48.300 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:48:48.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:48:48.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:48.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:49.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:50.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:50.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:50.389 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:48:50.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:48:50.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:50.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:51.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:48:52.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:48:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:48:52.551 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:48:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:48:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:48:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:48:53.063 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/175433,v1:127.0.0.1:6811/175433] [v2:127.0.0.1:6812/175433,v1:127.0.0.1:6813/175433] exists,up 96c6c3f0-782b-46ab-8ada-928c804c20c6 2024-09-15T02:48:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:48:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:48:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:48:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:48:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:48:53.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:48:53.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:48:53.070 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:48:53.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:48:53.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:48:53.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:48:53.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:48:53.072 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:48:53.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f00108c7-3f9b-49ad-8bda-055548c981d8 2024-09-15T02:48:53.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f00108c7-3f9b-49ad-8bda-055548c981d8' 2024-09-15T02:48:53.072 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 f00108c7-3f9b-49ad-8bda-055548c981d8 2024-09-15T02:48:53.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:48:53.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVS+ZmsQgpBRAAK3M0hTTqOB4IdncFHQqI/w== 2024-09-15T02:48:53.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVS+ZmsQgpBRAAK3M0hTTqOB4IdncFHQqI/w=="}' 2024-09-15T02:48:53.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f00108c7-3f9b-49ad-8bda-055548c981d8 -i td/test-erasure-eio/2/new.json 2024-09-15T02:48:53.564 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:48:53.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:48:53.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVS+ZmsQgpBRAAK3M0hTTqOB4IdncFHQqI/w== --osd-uuid f00108c7-3f9b-49ad-8bda-055548c981d8 2024-09-15T02:48:53.608 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:53.607+0000 7f7826d7f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:53.610 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:53.609+0000 7f7826d7f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:53.611 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:53.610+0000 7f7826d7f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:48:53.611 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:53.610+0000 7f7826d7f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:48:53.611 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:53.610+0000 7f7826d7f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:48:53.611 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:53.610+0000 7f7826d7f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:48:53.688 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:48:53.687+0000 7f7826d7f440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:48:59.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:48:59.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:48:59.653 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:48:59.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:48:59.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:49:00.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:49:00.144 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:49:00.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:49:00.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:49:00.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:49:00.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:49:00.174 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:00.173+0000 7fcbeaf1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:00.176 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:00.175+0000 7fcbeaf1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:00.177 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:00.176+0000 7fcbeaf1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:49:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:49:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:49:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:49:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:49:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:00.501 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:49:00.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:49:00.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:00.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:00.691 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:00.690+0000 7fcbeaf1c440 -1 Falling back to public interface 2024-09-15T02:49:00.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:01.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:01.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:01.858 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:49:01.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:49:01.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:01.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:02.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:03.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:03.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:03.224 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:49:03.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:49:03.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:03.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:03.280 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:03.279+0000 7fcbeaf1c440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:49:03.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:04.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:04.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:04.598 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:49:04.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:49:04.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:04.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:05.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:06.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:06.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:06.121 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:49:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:49:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:06.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:06.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:07.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:07.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:07.556 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:49:07.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:49:07.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:07.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:08.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:09.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:09.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:09.058 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:49:09.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:49:09.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:09.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:09.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:10.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:10.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:10.438 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:49:10.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:49:10.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:10.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:10.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:11.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:11.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:11.920 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:49:11.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:49:11.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:11.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:12.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:13.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:13.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:13.442 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:49:13.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:49:13.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:13.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:13.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:14.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:14.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:14.857 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:49:14.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:49:14.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:14.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:15.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:16.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:16.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:16.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:49:16.231 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:49:16.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:16.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:17.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:18.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:18.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:18.411 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:49:18.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:49:18.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:18.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:19.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:20.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:20.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:20.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:49:20.148 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:49:20.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:20.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:21.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:22.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:22.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:22.181 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:49:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:49:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:22.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:23.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:24.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:24.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:24.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:49:24.121 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:49:24.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:24.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:24.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:25.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:25.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:25.819 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:49:25.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:49:25.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:25.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:26.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:27.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:27.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:27.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:49:27.623 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:49:27.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:27.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:49:28.173 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/177162,v1:127.0.0.1:6819/177162] [v2:127.0.0.1:6820/177162,v1:127.0.0.1:6821/177162] exists,up f00108c7-3f9b-49ad-8bda-055548c981d8 2024-09-15T02:49:28.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:49:28.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:49:28.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:49:28.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:49:28.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:49:28.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae36a28c-2327-42f9-966d-9c18ba1f2c71 2024-09-15T02:49:28.181 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 ae36a28c-2327-42f9-966d-9c18ba1f2c71 2024-09-15T02:49:28.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ae36a28c-2327-42f9-966d-9c18ba1f2c71' 2024-09-15T02:49:28.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:49:28.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4S+ZmDDnACxAAcLjcATJOmItBN2VrOl04yw== 2024-09-15T02:49:28.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4S+ZmDDnACxAAcLjcATJOmItBN2VrOl04yw=="}' 2024-09-15T02:49:28.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae36a28c-2327-42f9-966d-9c18ba1f2c71 -i td/test-erasure-eio/3/new.json 2024-09-15T02:49:28.825 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:49:28.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:49:28.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4S+ZmDDnACxAAcLjcATJOmItBN2VrOl04yw== --osd-uuid ae36a28c-2327-42f9-966d-9c18ba1f2c71 2024-09-15T02:49:28.870 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:28.868+0000 7f4164b66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:28.871 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:28.870+0000 7f4164b66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:28.873 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:28.871+0000 7f4164b66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:28.873 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:28.871+0000 7f4164b66440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:49:28.873 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:28.871+0000 7f4164b66440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:49:28.873 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:28.871+0000 7f4164b66440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:49:28.949 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:28.947+0000 7f4164b66440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:49:34.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:49:34.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:49:34.798 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:49:34.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:49:34.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:49:35.290 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:49:35.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:49:35.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:49:35.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:49:35.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:49:35.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:49:35.321 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:35.320+0000 7f1712617440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:35.348 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:35.347+0000 7f1712617440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:35.349 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:35.349+0000 7f1712617440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:49:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:49:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:49:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:49:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:49:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:49:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:49:35.738 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:49:35.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:35.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:35.878 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:35.877+0000 7f1712617440 -1 Falling back to public interface 2024-09-15T02:49:36.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:37.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:37.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:37.098 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:49:37.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:49:37.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:37.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:37.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:38.412 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:49:38.411+0000 7f1712617440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:49:38.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:38.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:38.460 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:49:38.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:49:38.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:38.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:38.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:39.961 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:49:39.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:39.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:39.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:49:39.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:39.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:40.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:41.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:41.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:41.613 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:49:41.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:49:41.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:41.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:41.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:42.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:42.986 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:49:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:49:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:42.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:43.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:44.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:44.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:44.502 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:49:44.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:49:44.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:44.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:44.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:45.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:45.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:45.939 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:49:45.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:49:45.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:45.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:46.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:47.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:47.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:47.473 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:49:47.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:49:47.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:47.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:47.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:48.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:48.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:48.880 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:49:48.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:49:48.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:48.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:49.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:50.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:50.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:50.396 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:49:50.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:49:50.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:50.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:50.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:51.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:51.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:51.758 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:49:51.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:49:51.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:51.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:52.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:53.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:53.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:53.126 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:49:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:49:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:53.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:53.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:54.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:54.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:54.903 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:49:54.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:49:54.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:54.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:56.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:57.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:57.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:57.004 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:49:57.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:49:57.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:57.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:57.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:49:58.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:49:58.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:49:58.446 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:49:58.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:49:58.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:49:58.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:49:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:00.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:00.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:00.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:50:00.330 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:50:00.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:00.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:50:00.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:01.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:01.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:01.774 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:50:01.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:50:01.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:01.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:50:02.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:03.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:03.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:03.255 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:50:03.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:50:03.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:03.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:50:03.745 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/178916,v1:127.0.0.1:6827/178916] [v2:127.0.0.1:6828/178916,v1:127.0.0.1:6829/178916] exists,up ae36a28c-2327-42f9-966d-9c18ba1f2c71 2024-09-15T02:50:03.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:50:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:50:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:50:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:50:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T02:50:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:50:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:50:03.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T02:50:03.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T02:50:03.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:50:03.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:50:03.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:50:03.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:50:03.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:50:03.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:50:03.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:50:03.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:50:03.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T02:50:03.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:50:03.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=042edf95-e88c-4e13-ae98-cecfde9965e2 2024-09-15T02:50:03.757 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 042edf95-e88c-4e13-ae98-cecfde9965e2 2024-09-15T02:50:03.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 042edf95-e88c-4e13-ae98-cecfde9965e2' 2024-09-15T02:50:03.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:50:03.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbS+ZmjefJLRAAoMF7Whfy9uJY8xI6xCkjqQ== 2024-09-15T02:50:03.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbS+ZmjefJLRAAoMF7Whfy9uJY8xI6xCkjqQ=="}' 2024-09-15T02:50:03.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 042edf95-e88c-4e13-ae98-cecfde9965e2 -i td/test-erasure-eio/4/new.json 2024-09-15T02:50:04.370 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:50:04.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T02:50:04.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbS+ZmjefJLRAAoMF7Whfy9uJY8xI6xCkjqQ== --osd-uuid 042edf95-e88c-4e13-ae98-cecfde9965e2 2024-09-15T02:50:04.414 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:04.413+0000 7fe8c2212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:04.480 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:04.479+0000 7fe8c2212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:04.481 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:04.480+0000 7fe8c2212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:04.481 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:04.480+0000 7fe8c2212440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:50:04.481 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:04.480+0000 7fe8c2212440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:50:04.481 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:04.480+0000 7fe8c2212440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:50:04.590 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:04.590+0000 7fe8c2212440 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T02:50:11.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T02:50:11.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:50:11.556 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:50:11.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:50:11.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:50:12.052 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:50:12.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:50:12.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:50:12.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:50:12.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:50:12.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:50:12.084 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:12.082+0000 7f37c5c73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:12.111 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:12.110+0000 7f37c5c73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:12.112 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:12.111+0000 7f37c5c73440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:50:12.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:50:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:50:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:50:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:50:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:50:12.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:12.447 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:50:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:50:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:12.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:12.653 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:12.653+0000 7f37c5c73440 -1 Falling back to public interface 2024-09-15T02:50:12.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:13.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:13.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:13.810 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:50:13.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:50:13.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:13.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:14.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:15.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:15.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:15.178 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:50:15.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:50:15.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:15.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:15.224 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:50:15.223+0000 7f37c5c73440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:50:15.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:16.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:16.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:16.542 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:50:16.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:50:16.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:16.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:16.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:17.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:17.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:17.980 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:50:17.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:50:17.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:17.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:18.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:19.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:19.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:19.455 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:50:19.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:50:19.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:19.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:19.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:20.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:20.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:20.824 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:50:20.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:50:20.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:20.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:21.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:22.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:22.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:22.381 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:50:22.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:50:22.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:22.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:22.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:23.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:23.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:23.746 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:50:23.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:50:23.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:23.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:24.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:25.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:25.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:25.166 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:50:25.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:50:25.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:25.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:25.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:26.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:26.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:26.534 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:50:26.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:50:26.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:26.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:26.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:27.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:27.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:27.975 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:50:27.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:50:27.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:27.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:28.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:29.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:29.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:29.586 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:50:29.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:50:29.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:29.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:30.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:31.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:31.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:31.342 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:50:31.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:50:31.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:31.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:32.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:33.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:33.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:33.120 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:50:33.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:50:33.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:33.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:33.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:34.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:34.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:34.864 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:50:34.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:50:34.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:34.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:35.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:36.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:36.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:36.712 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:50:36.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:50:36.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:36.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:37.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:38.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:38.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:38.492 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:50:38.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:50:38.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:38.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:39.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:50:40.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:50:40.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:50:40.383 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:50:40.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:50:40.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:50:40.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:50:40.769 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/180695,v1:127.0.0.1:6835/180695] [v2:127.0.0.1:6836/180695,v1:127.0.0.1:6837/180695] exists,up 042edf95-e88c-4e13-ae98-cecfde9965e2 2024-09-15T02:50:40.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:50:40.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:50:40.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:50:40.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:50:40.844 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:50:40.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:50:40.855 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:47:53.432+0000 7fef9c20a440 0 load: jerasure load: lrc 2024-09-15T02:50:40.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:50:40.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-09-15T02:50:40.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T02:50:40.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:50:40.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:50:40.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T02:50:40.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:50:40.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T02:50:40.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:50:40.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T02:50:41.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:50:41.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:50:43.531 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:50:43.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:50:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:50:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:50:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:50:44.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:50:44.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:50:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:50:44.545 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:50:44.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:50:44.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:50:44.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:50:44.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:50:44.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:50:44.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:50:44.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:50:44.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:50:44.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:45.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:50:45.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803810 2024-09-15T02:50:45.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803810 2024-09-15T02:50:45.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810' 2024-09-15T02:50:45.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:45.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:50:45.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574877 2024-09-15T02:50:45.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574877 2024-09-15T02:50:45.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574877' 2024-09-15T02:50:45.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:45.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:50:45.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378644 2024-09-15T02:50:45.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378644 2024-09-15T02:50:45.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574877 2-81604378644' 2024-09-15T02:50:45.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:45.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:50:45.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084301 2024-09-15T02:50:45.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084301 2024-09-15T02:50:45.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574877 2-81604378644 3-120259084301' 2024-09-15T02:50:45.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:50:45.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:50:45.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691847 2024-09-15T02:50:45.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691847 2024-09-15T02:50:45.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574877 2-81604378644 3-120259084301 4-171798691847' 2024-09-15T02:50:45.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:45.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803810 2024-09-15T02:50:45.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:45.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:50:45.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803810 2024-09-15T02:50:45.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:45.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803810 2024-09-15T02:50:45.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-09-15T02:50:45.806 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803810 2024-09-15T02:50:45.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:50:46.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803807 -lt 25769803810 2024-09-15T02:50:46.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:50:47.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:50:47.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:50:47.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803810 2024-09-15T02:50:47.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:47.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574877 2024-09-15T02:50:47.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:47.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:50:47.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574877 2024-09-15T02:50:47.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:47.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574877 2024-09-15T02:50:47.522 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574877 2024-09-15T02:50:47.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574877' 2024-09-15T02:50:47.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:50:47.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574877 -lt 55834574877 2024-09-15T02:50:47.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:47.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378644 2024-09-15T02:50:47.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:47.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:50:47.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378644 2024-09-15T02:50:47.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:47.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378644 2024-09-15T02:50:47.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-09-15T02:50:47.893 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378644 2024-09-15T02:50:47.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:50:48.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378645 -lt 81604378644 2024-09-15T02:50:48.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:48.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084301 2024-09-15T02:50:48.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:48.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:50:48.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084301 2024-09-15T02:50:48.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:48.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084301 2024-09-15T02:50:48.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084301' 2024-09-15T02:50:48.318 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084301 2024-09-15T02:50:48.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:50:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084302 -lt 120259084301 2024-09-15T02:50:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:50:48.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691847 2024-09-15T02:50:48.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:50:48.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:50:48.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691847 2024-09-15T02:50:48.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:50:48.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691847 2024-09-15T02:50:48.707 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 171798691847 2024-09-15T02:50:48.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691847' 2024-09-15T02:50:48.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:50:49.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691847 -lt 171798691847 2024-09-15T02:50:49.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:50:49.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:50:49.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:50:49.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:50:49.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:50:49.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:50:49.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:50:49.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:50:49.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:50:49.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:50:49.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:50:49.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:50:49.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:50:49.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:50:49.865 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:50:50.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:50:50.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:50:50.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:50:50.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:45.061222+0000 0'0 47:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-15T02:50:42.956359+0000 0'0 2024-09-15T02:50:42.956359+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:17:42.882056+0000 0 0 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.999165+0000 0'0 47:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:00:49.512163+0000 0 0 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.785599+0000 0'0 47:105 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:12:38.637245+0000 0 0 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.957125+0000 0'0 47:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T08:50:29.221265+0000 0 0 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:49:27.860623+0000 0'0 47:43 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T03:16:15.455315+0000 0 0 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-15T02:50:50.699 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T02:50:50.700 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-15T02:50:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:50:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:50:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:50:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T02:50:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:50:50.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:50:50.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:50:50.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:50:50.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:50:50.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:50:50.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:50:50.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:50:50.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:50.891 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-09-15T02:50:50.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:50:50.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:50:50.892 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:50:50.892 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:50:51.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:50:51.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:50:51.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:50:51.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:50:52.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:50:52.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2024-09-15T02:50:53.018 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already down. 2024-09-15T02:50:53.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2024-09-15T02:50:53.517 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2024-09-15T02:50:53.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-15T02:50:53.860 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 active+undersized+degraded 2024-09-15T02:50:52.614245+0000 48'1 50:24 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-15T02:50:42.956359+0000 0'0 2024-09-15T02:50:42.956359+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:21:52.548712+0000 0 0 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.999165+0000 0'0 47:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:00:49.512163+0000 0 0 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.785599+0000 0'0 47:105 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:12:38.637245+0000 0 0 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.957125+0000 0'0 47:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T08:50:29.221265+0000 0 0 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-15T02:50:52.538641+0000 0'0 50:52 [1,0] 1 [1,0] 1 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T03:13:18.924388+0000 0 0 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T02:50:53.861 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-15T02:50:53.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-15T02:50:53.872 INFO:tasks.workunit.client.0.smithi130.stderr:4+0 records in 2024-09-15T02:50:53.872 INFO:tasks.workunit.client.0.smithi130.stderr:4+0 records out 2024-09-15T02:50:53.872 INFO:tasks.workunit.client.0.smithi130.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.5658e-05 s, 47.8 MB/s 2024-09-15T02:50:53.872 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2024-09-15T02:50:53.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:53.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:54.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:54.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:54.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:54.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:54.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:54.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:55.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:55.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:55.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:55.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:55.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:55.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:55.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:55.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:56.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:56.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:56.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:56.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:56.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:56.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:56.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:56.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:57.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:57.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:57.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:57.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:57.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:57.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:57.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:57.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:58.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:58.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:58.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:58.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:58.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:58.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:58.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:58.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:59.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:59.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:59.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:59.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:59.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:59.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:59.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:59.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-15T02:50:59.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:50:59.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:00.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:00.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:00.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:00.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:00.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:00.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:00.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:01.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:01.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:01.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:01.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:01.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:01.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:02.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:02.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:02.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:02.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:02.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:02.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:02.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:02.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:02.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:02.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:03.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:03.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:03.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:03.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:03.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:03.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:03.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:03.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:04.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:04.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:04.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:04.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:04.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:04.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:04.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:04.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:05.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:05.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:05.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:05.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:05.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:05.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:05.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:06.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:06.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:06.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:06.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:06.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:06.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:06.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:06.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:06.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:06.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:07.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:07.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:07.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:07.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:07.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:07.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:07.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:07.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:08.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:08.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:08.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:08.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:08.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:08.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:08.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:08.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:09.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:09.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:09.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:09.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:09.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:09.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:09.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:09.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:10.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:10.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:10.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:10.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:10.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:10.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:10.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:11.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:11.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:11.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:11.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:11.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:11.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:11.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:11.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:11.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:11.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:12.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:12.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:12.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:12.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:12.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:12.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:12.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:13.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:13.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:13.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:13.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:13.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:13.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:13.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:13.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:14.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:14.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:14.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:14.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:14.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:14.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:14.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:14.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:15.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:15.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:15.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:15.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:15.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:15.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:15.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:16.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:16.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:16.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:16.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:16.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:16.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:16.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:16.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:17.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:17.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:17.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:17.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:17.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:17.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:17.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:17.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:18.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:18.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-15T02:51:18.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-09-15T02:51:18.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:51:18.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:51:18.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:51:18.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:51:18.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:51:18.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj75 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:51:18.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-15T02:51:18.318 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-15T02:51:18.318 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:51:18.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:51:18.683 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:51:18.683 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:51:18.683 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:51:18.683 INFO:tasks.workunit.client.0.smithi130.stderr:2147483647' 2024-09-15T02:51:18.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-15T02:51:18.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T02:51:18.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:51:18.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T02:51:18.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:51:18.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:51:18.686 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:51:18.687 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:51:18.687 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:18.687 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:18.687 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:51:18.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:51:18.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:51:18.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:51:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-09-15T02:51:18.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-09-15T02:51:18.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:51:18.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj75 2024-09-15T02:51:18.852 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:51:18.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-15T02:51:18.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-15T02:51:18.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:2147483647' 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:51:19.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T02:51:19.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:51:19.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T02:51:19.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T02:51:19.205 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:51:19.206 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:19.206 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:19.206 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T02:51:19.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:51:19.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:51:19.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:51:19.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-09-15T02:51:19.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-09-15T02:51:19.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:51:19.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:51:19.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:51:19.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:51:19.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:51:19.373 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:51:19.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T02:51:19.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:51:19.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T02:51:19.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T02:51:19.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:51:19.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:51:19.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:51:19.392 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:19.390+0000 7f6a9851d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:19.434 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:19.432+0000 7f6a9851d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:19.435 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:19.434+0000 7f6a9851d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:19.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T02:51:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:51:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:51:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:51:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:51:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:19.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:51:19.724 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:51:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:51:19.953 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:19.951+0000 7f6a9851d440 -1 Falling back to public interface 2024-09-15T02:51:20.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:21.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:21.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:21.095 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:51:21.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:51:21.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:21.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:51:21.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:22.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:22.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:22.453 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:51:22.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:51:22.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:22.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:51:22.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:51:23.259 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:23.257+0000 7f6a9851d440 -1 osd.2 48 log_to_monitors true 2024-09-15T02:51:23.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:51:23.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:51:23.871 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:51:23.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:51:23.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:51:23.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:51:24.301 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up out weight 0 up_from 55 up_thru 19 down_at 49 last_clean_interval [19,48) [v2:127.0.0.1:6818/184365,v1:127.0.0.1:6819/184365] [v2:127.0.0.1:6820/184365,v1:127.0.0.1:6821/184365] exists,up f00108c7-3f9b-49ad-8bda-055548c981d8 2024-09-15T02:51:24.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:51:24.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:51:24.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:51:24.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2024-09-15T02:51:25.488 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2024-09-15T02:51:25.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2024-09-15T02:51:40.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2024-09-15T02:51:40.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-09-15T02:51:40.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2024-09-15T02:51:40.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-15T02:51:40.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-15T02:51:40.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-15T02:51:40.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T02:51:40.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-15T02:51:40.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-15T02:51:40.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2024-09-15T02:51:40.857 INFO:tasks.workunit.client.0.smithi130.stdout:active+recovery_unfound+degraded 2024-09-15T02:51:40.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-09-15T02:51:40.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2024-09-15T02:51:40.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-15T02:51:41.186 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-15T02:51:28.989614+0000 53'101 59:346 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-15T02:50:42.956359+0000 0'0 2024-09-15T02:50:42.956359+0000 0 0 periodic scrub scheduled @ 2024-09-16T03:08:39.738328+0000 0 0 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:51:26.312590+0000 0'0 59:51 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:30:18.434462+0000 0 0 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.785599+0000 0'0 58:116 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:12:38.637245+0000 0 0 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:50:42.957125+0000 0'0 58:24 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T08:50:29.221265+0000 0 0 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T02:51:26.312352+0000 0'0 59:80 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T02:47:44.724380+0000 0'0 2024-09-15T02:47:44.724380+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:33:49.708794+0000 0 0 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T02:51:41.187 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-15T02:51:41.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:51:41.297 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-15T02:51:41.297 INFO:tasks.workunit.client.0.smithi130.stdout: "num_missing": 1, 2024-09-15T02:51:41.297 INFO:tasks.workunit.client.0.smithi130.stdout: "num_unfound": 1, 2024-09-15T02:51:41.297 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": [ 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "oid": { 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "oid": "obj75", 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "key": "", 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "snapid": -2, 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "hash": 3565005477, 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "max": 0, 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "pool": 2, 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "namespace": "" 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.298 INFO:tasks.workunit.client.0.smithi130.stdout: "need": "53'76", 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: "have": "0'0", 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: "flags": "none", 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: "locations": [ 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.299 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.300 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "NotRecovering", 2024-09-15T02:51:41.300 INFO:tasks.workunit.client.0.smithi130.stdout: "available_might_have_unfound": true, 2024-09-15T02:51:41.300 INFO:tasks.workunit.client.0.smithi130.stdout: "might_have_unfound": [], 2024-09-15T02:51:41.300 INFO:tasks.workunit.client.0.smithi130.stdout: "more": false 2024-09-15T02:51:41.300 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-15T02:51:41.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trimq": "[]", 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "snap_trimq_len": 0, 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+recovery_unfound+degraded", 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": 59, 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_recovery_backfill": [ 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:51:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: "2(4)", 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: "info": { 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s0", 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'101", 2024-09-15T02:51:41.410 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'75", 2024-09-15T02:51:41.411 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "0'0", 2024-09-15T02:51:41.411 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 101, 2024-09-15T02:51:41.411 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:51:41.411 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:51:41.411 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:51:41.411 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 46, 2024-09-15T02:51:41.411 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 46, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 57, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 46, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 57, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 57, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 46, 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.412 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'101", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 347, 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 59, 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+recovery_unfound+degraded", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:51:29.040748+0000", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:51:28.989614+0000", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:51:29.040748+0000", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:51:29.040748+0000", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.413 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:51:26.312431+0000", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:51:26.312431+0000", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:51:29.040748+0000", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:51:25.914622+0000", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:51:29.040748+0000", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 57, 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "0'0", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "0'0", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 46, 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:51:41.414 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:51:41.415 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 0, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:51:41.416 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:51:41.417 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:51:41.417 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T03:08:39.738328+0000", 2024-09-15T02:51:41.417 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:51:41.417 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:51:41.417 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:51:41.417 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 413696, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 505, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 1, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 3, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:51:41.418 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 1, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 404, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:51:41.419 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 99, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 405504, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:51:41.420 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:51:41.421 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:51:41.422 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:51:41.422 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:51:41.422 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:51:41.422 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:51:41.423 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.423 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:51:41.423 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.423 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.423 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.423 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.424 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.424 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.424 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:51:41.424 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.424 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.424 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.424 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.425 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.425 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.425 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:51:41.425 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:51:41.425 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:51:41.425 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.425 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:51:41.426 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 100 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),4(2)", 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 1 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:51:41.427 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.428 INFO:tasks.workunit.client.0.smithi130.stdout: "peer_info": [ 2024-09-15T02:51:41.429 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.429 INFO:tasks.workunit.client.0.smithi130.stdout: "peer": "0(3)", 2024-09-15T02:51:41.429 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s3", 2024-09-15T02:51:41.429 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'101", 2024-09-15T02:51:41.429 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'101", 2024-09-15T02:51:41.429 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "0'0", 2024-09-15T02:51:41.429 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 101, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 46, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 46, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 57, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 46, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 57, 2024-09-15T02:51:41.430 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 57, 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 46, 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'101", 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 128, 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 53, 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2024-09-15T02:51:41.431 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.432 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.432 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.432 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.432 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.432 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:50:52.287363+0000", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:50:52.287205+0000", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 57, 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "0'0", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "0'0", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 46, 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.433 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 0, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:51:41.434 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T10:21:52.548712+0000", 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 413696, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 505, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T02:51:41.435 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 101, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 404, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:51:41.436 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:51:41.437 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.438 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:51:41.439 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 101 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.440 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "peer": "1(1)", 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s1", 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'101", 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'101", 2024-09-15T02:51:41.441 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "0'0", 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 101, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 46, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 46, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 57, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 46, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 57, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 57, 2024-09-15T02:51:41.442 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 46, 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'101", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 128, 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 53, 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.443 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:50:52.287363+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:50:52.287205+0000", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 57, 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "0'0", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "0'0", 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 46, 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.444 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 0, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:51:41.445 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T10:21:52.548712+0000", 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:51:41.446 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 413696, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 505, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 1, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 101, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:51:41.447 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 404, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:51:41.448 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:51:41.449 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.450 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 101 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:51:41.451 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.452 INFO:tasks.workunit.client.0.smithi130.stdout: "peer": "2(4)", 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s4", 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'101", 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "48'1", 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "0'0", 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 1, 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 46, 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 46, 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 57, 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.453 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 46, 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 57, 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 57, 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 46, 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:51:41.454 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "48'1", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 14, 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 48, 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+clean", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:50:45.061222+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:50:45.060782+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:50:45.060782+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.455 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 57, 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "0'0", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "0'0", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 46, 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 1, 2024-09-15T02:51:41.456 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 0, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 1, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T13:17:42.882056+0000", 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:51:41.457 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 4096, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 1, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 5, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 1, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 1, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:51:41.458 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 1, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 4, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:51:41.459 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:51:41.460 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.461 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [], 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [], 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:51:41.462 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "peer": "4(2)", 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "pgid": "2.0s2", 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "last_update": "53'101", 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "last_complete": "53'101", 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "log_tail": "0'0", 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "last_user_version": 101, 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill": "MAX", 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [], 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "history": { 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_created": 46, 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch_pool_created": 46, 2024-09-15T02:51:41.463 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_started": 57, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_interval_clean": 46, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_split": 0, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_marked_full": 0, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "same_up_since": 57, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "same_interval_since": 57, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "same_primary_since": 46, 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.464 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_readable_until_ub": 0 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "stats": { 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "53'101", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_seq": 128, 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "reported_epoch": 53, 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "active+undersized+degraded", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fresh": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "last_change": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "last_active": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "last_peered": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean": "2024-09-15T02:50:50.889009+0000", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_active": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.465 INFO:tasks.workunit.client.0.smithi130.stdout: "last_became_peered": "2024-09-15T02:50:52.614245+0000", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_unstale": "2024-09-15T02:51:18.312361+0000", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_undegraded": "2024-09-15T02:50:52.287363+0000", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_fullsized": "2024-09-15T02:50:52.287205+0000", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "mapping_epoch": 57, 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "log_start": "0'0", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_start": "0'0", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "created": 46, 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_clean": 47, 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "parent": "0.0", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "parent_split_bits": 0, 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub": "0'0", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub": "0'0", 2024-09-15T02:51:41.466 INFO:tasks.workunit.client.0.smithi130.stdout: "last_deep_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "last_clean_scrub_stamp": "2024-09-15T02:50:42.956359+0000", 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_scrubbed": 0, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "log_size": 101, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "log_dups_size": 0, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "ondisk_log_size": 101, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "stats_invalid": false, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "dirty_stats_invalid": false, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "omap_stats_invalid": false, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_stats_invalid": false, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "pin_stats_invalid": false, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "manifest_stats_invalid": false, 2024-09-15T02:51:41.467 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrimq_len": 0, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "last_scrub_duration": 0, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T10:21:52.548712+0000", 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_duration": 0, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "objects_trimmed": 0, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "snaptrim_duration": 0, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "stat_sum": { 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes": 413696, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects": 101, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_clones": 0, 2024-09-15T02:51:41.468 INFO:tasks.workunit.client.0.smithi130.stdout: "num_object_copies": 505, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_missing": 0, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_degraded": 101, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_misplaced": 0, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_unfound": 0, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_dirty": 101, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_whiteouts": 0, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read": 0, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_read_kb": 0, 2024-09-15T02:51:41.469 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write": 101, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_write_kb": 404, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_scrub_errors": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_deep_scrub_errors": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_recovered": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_recovered": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_keys_recovered": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_omap": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_kb": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict": 0, 2024-09-15T02:51:41.470 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_kb": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_promote": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_high": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_flush_mode_low": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_some": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_evict_mode_full": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_pinned": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_legacy_snapsets": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_large_omap_objects": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_manifest": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_bytes": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_omap_keys": 0, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "num_objects_repaired": 0 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.471 INFO:tasks.workunit.client.0.smithi130.stdout: "up": [ 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: "acting": [ 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 3, 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 1, 2024-09-15T02:51:41.472 INFO:tasks.workunit.client.0.smithi130.stdout: 4, 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: 0, 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: 2 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: "avail_no_missing": [ 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: "3(0)", 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: "0(3)", 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: "1(1)", 2024-09-15T02:51:41.473 INFO:tasks.workunit.client.0.smithi130.stdout: "4(2)" 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: "object_location_counts": [ 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": 101 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: "blocked_by": [], 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: "up_primary": 3, 2024-09-15T02:51:41.474 INFO:tasks.workunit.client.0.smithi130.stdout: "acting_primary": 3, 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "purged_snaps": [] 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "empty": 0, 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "dne": 0, 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "incomplete": 0, 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "last_epoch_started": 59, 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "hit_set_history": { 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "current_last_update": "0'0", 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [] 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_state": [ 2024-09-15T02:51:41.475 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.476 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "Started/Primary/Active", 2024-09-15T02:51:41.476 INFO:tasks.workunit.client.0.smithi130.stdout: "enter_time": "2024-09-15T02:51:25.913770+0000", 2024-09-15T02:51:41.476 INFO:tasks.workunit.client.0.smithi130.stdout: "might_have_unfound": [ 2024-09-15T02:51:41.476 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.476 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "0(3)", 2024-09-15T02:51:41.476 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "already probed" 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "1(1)", 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "already probed" 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "2(4)", 2024-09-15T02:51:41.477 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "already probed" 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: "osd": "4(2)", 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: "status": "already probed" 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_progress": { 2024-09-15T02:51:41.478 INFO:tasks.workunit.client.0.smithi130.stdout: "backfill_targets": [], 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "waiting_on_backfill": [], 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "last_backfill_started": "MIN", 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "backfill_info": { 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "begin": "MIN", 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "end": "MIN", 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "objects": [] 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "peer_backfill_info": [], 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "backfills_in_flight": [], 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "recovering": [], 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_backend": { 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "recovery_ops": [], 2024-09-15T02:51:41.479 INFO:tasks.workunit.client.0.smithi130.stdout: "read_ops": [] 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "name": "Started", 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "enter_time": "2024-09-15T02:51:25.254008+0000" 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "scrubber": { 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "active": false, 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "must_scrub": false, 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "must_deep_scrub": false, 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "must_repair": false, 2024-09-15T02:51:41.480 INFO:tasks.workunit.client.0.smithi130.stdout: "need_auto": false, 2024-09-15T02:51:41.481 INFO:tasks.workunit.client.0.smithi130.stdout: "scrub_reg_stamp": "2024-09-16T03:08:39.738328+0000", 2024-09-15T02:51:41.481 INFO:tasks.workunit.client.0.smithi130.stdout: "schedule": "scrub scheduled @ 2024-09-16T03:08:39.738328+0000" 2024-09-15T02:51:41.481 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-15T02:51:41.481 INFO:tasks.workunit.client.0.smithi130.stdout: "agent_state": {} 2024-09-15T02:51:41.481 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-15T02:51:41.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:51:41.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2024-09-15T02:51:41.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:51:41.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-09-15T02:51:41.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2024-09-15T02:51:41.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2024-09-15T02:51:41.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T02:51:41.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-09-15T02:51:41.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2024-09-15T02:51:41.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2024-09-15T02:51:41.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-15T02:51:46.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2024-09-15T02:51:46.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-15T02:51:46.993 INFO:tasks.workunit.client.0.smithi130.stderr:pg has no unfound objects 2024-09-15T02:51:47.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:51:47.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:51:47.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:51:47.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:51:47.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:51:47.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:51:47.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:51:47.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:51:47.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:51:47.444 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:51:47.444 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:51:47.444 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:51:47.444 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-15T02:51:47.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:51:47.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:51:47.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:51:47.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803824 2024-09-15T02:51:47.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803824 2024-09-15T02:51:47.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824' 2024-09-15T02:51:47.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:51:47.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:51:47.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574891 2024-09-15T02:51:47.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574891 2024-09-15T02:51:47.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891' 2024-09-15T02:51:47.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:51:47.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:51:47.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201288 2024-09-15T02:51:47.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201288 2024-09-15T02:51:47.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-236223201288' 2024-09-15T02:51:47.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:51:47.774 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:51:47.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084316 2024-09-15T02:51:47.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084316 2024-09-15T02:51:47.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-236223201288 3-120259084316' 2024-09-15T02:51:47.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:51:47.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:51:47.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691861 2024-09-15T02:51:47.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691861 2024-09-15T02:51:47.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574891 2-236223201288 3-120259084316 4-171798691861' 2024-09-15T02:51:47.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:51:47.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803824 2024-09-15T02:51:47.984 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:51:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:51:47.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803824 2024-09-15T02:51:47.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:51:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803824 2024-09-15T02:51:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-09-15T02:51:47.987 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803824 2024-09-15T02:51:47.987 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:51:48.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803824 -lt 25769803824 2024-09-15T02:51:48.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:51:48.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574891 2024-09-15T02:51:48.394 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:51:48.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:51:48.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574891 2024-09-15T02:51:48.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:51:48.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574891 2024-09-15T02:51:48.396 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574891 2024-09-15T02:51:48.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574891' 2024-09-15T02:51:48.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:51:48.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574892 -lt 55834574891 2024-09-15T02:51:48.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:51:48.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-236223201288 2024-09-15T02:51:48.783 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:51:48.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:51:48.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-236223201288 2024-09-15T02:51:48.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:51:48.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201288 2024-09-15T02:51:48.785 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 236223201288 2024-09-15T02:51:48.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 236223201288' 2024-09-15T02:51:48.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:51:49.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201288 -lt 236223201288 2024-09-15T02:51:49.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:51:49.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084316 2024-09-15T02:51:49.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:51:49.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:51:49.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084316 2024-09-15T02:51:49.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:51:49.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084316 2024-09-15T02:51:49.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084316' 2024-09-15T02:51:49.139 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084316 2024-09-15T02:51:49.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:51:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084316 -lt 120259084316 2024-09-15T02:51:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:51:49.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691861 2024-09-15T02:51:49.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:51:49.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:51:49.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691861 2024-09-15T02:51:49.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:51:49.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691861 2024-09-15T02:51:49.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691861' 2024-09-15T02:51:49.498 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 171798691861 2024-09-15T02:51:49.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:51:49.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691861 -lt 171798691861 2024-09-15T02:51:49.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:51:49.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:51:49.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:51:50.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:51:50.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:51:50.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:51:50.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:51:50.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:51:51.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:51:51.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:51:51.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:51:51.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2024-09-15T02:51:51.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-15T02:51:51.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-15T02:51:51.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-15T02:51:51.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-15T02:51:51.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-15T02:51:51.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-15T02:51:51.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-15T02:51:51.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-15T02:51:51.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-15T02:51:51.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-15T02:51:51.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-15T02:51:51.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-15T02:51:51.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-15T02:51:51.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-15T02:51:51.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-15T02:51:51.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-15T02:51:51.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-15T02:51:51.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-15T02:51:51.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-15T02:51:51.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-15T02:51:51.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-15T02:51:51.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-15T02:51:51.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-15T02:51:51.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-15T02:51:51.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-15T02:51:51.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-15T02:51:51.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-15T02:51:51.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-15T02:51:51.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:51.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:51.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-15T02:51:51.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-15T02:51:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-15T02:51:52.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-15T02:51:52.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-15T02:51:52.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-15T02:51:52.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-15T02:51:52.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-15T02:51:52.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-15T02:51:52.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-15T02:51:52.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-15T02:51:52.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-15T02:51:52.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-15T02:51:52.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-15T02:51:52.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-15T02:51:52.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-15T02:51:52.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-15T02:51:52.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-15T02:51:52.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-15T02:51:52.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-15T02:51:52.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-15T02:51:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-15T02:51:52.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-15T02:51:52.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-15T02:51:52.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-15T02:51:52.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-15T02:51:52.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-15T02:51:52.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-15T02:51:52.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-15T02:51:52.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-15T02:51:52.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-15T02:51:52.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-15T02:51:52.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-15T02:51:52.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-15T02:51:52.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-15T02:51:52.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:52.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:52.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-15T02:51:52.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-15T02:51:53.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-15T02:51:53.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-15T02:51:53.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-15T02:51:53.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-15T02:51:53.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-15T02:51:53.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-15T02:51:53.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-15T02:51:53.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-15T02:51:53.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-15T02:51:53.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-15T02:51:53.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-15T02:51:53.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-15T02:51:53.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.360 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-09-15T02:51:53.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-15T02:51:53.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-15T02:51:53.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-15T02:51:53.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-15T02:51:53.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-15T02:51:53.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-15T02:51:53.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-15T02:51:53.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-15T02:51:53.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-15T02:51:53.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-15T02:51:53.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-15T02:51:53.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-15T02:51:53.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-15T02:51:53.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-15T02:51:53.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-15T02:51:53.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-15T02:51:53.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-15T02:51:53.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-15T02:51:53.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-15T02:51:53.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-15T02:51:53.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-15T02:51:53.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-15T02:51:53.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-15T02:51:53.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-15T02:51:53.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-15T02:51:53.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:53.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T02:51:53.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-15T02:51:53.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-15T02:51:54.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:54.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-15T02:51:54.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-15T02:51:54.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:51:54.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:51:54.565 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:51:54.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T02:51:55.277 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T02:51:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T02:51:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:51:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:51:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:51:55.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:51:55.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:51:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:51:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:51:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:51:55.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:51:55.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:51:55.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:51:55.401 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:51:55.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:51:55.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:51:55.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:51:55.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:51:55.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:51:55.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:51:55.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:51:55.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:55.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:51:55.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:51:55.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:51:55.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:55.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:55.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:51:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:51:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:51:55.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:51:55.442 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:51:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:51:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:51:55.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:51:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:51:55.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:51:55.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:51:55.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:51:55.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:51:55.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:51:55.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:51:55.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:51:55.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:51:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:51:55.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:51:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:51:55.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:51:55.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:51:55.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:51:55.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:55.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:55.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:51:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:51:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:51:55.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:51:55.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:51:55.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:55.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:55.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:51:55.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:51:55.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:51:55.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:51:56.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:51:56.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:51:56.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:51:56.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:51:56.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:51:56.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:51:56.028 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:56.029 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:56.029 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:51:56.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:51:56.029 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:51:56.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:51:56.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:51:56.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:51:56.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:51:56.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:51:56.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:51:56.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:51:56.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:51:56.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:51:56.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:51:56.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:51:57.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:51:57.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:51:57.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:51:57.282 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:51:57.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:51:58.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:51:58.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:51:58.364 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:51:58.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:51:58.370 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:51:55.828+0000 7f3259663900 0 load: jerasure load: lrc 2024-09-15T02:51:58.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-09-15T02:51:58.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-09-15T02:51:58.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2024-09-15T02:51:58.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2024-09-15T02:51:58.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-15T02:51:58.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:51:58.372 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-15T02:51:58.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-15T02:51:58.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:51:58.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:51:58.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:51:58.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:51:58.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:51:58.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:51:58.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4684e24c-c8d5-4d53-9efa-12a2f95cf773 2024-09-15T02:51:58.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4684e24c-c8d5-4d53-9efa-12a2f95cf773' 2024-09-15T02:51:58.380 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 4684e24c-c8d5-4d53-9efa-12a2f95cf773 2024-09-15T02:51:58.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:51:58.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOS+ZmI+NTFxAAaBlXKvJZqB6/JH8PXokkiQ== 2024-09-15T02:51:58.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOS+ZmI+NTFxAAaBlXKvJZqB6/JH8PXokkiQ=="}' 2024-09-15T02:51:58.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4684e24c-c8d5-4d53-9efa-12a2f95cf773 -i td/test-erasure-eio/0/new.json 2024-09-15T02:51:58.660 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:51:58.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:51:58.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOS+ZmI+NTFxAAaBlXKvJZqB6/JH8PXokkiQ== --osd-uuid 4684e24c-c8d5-4d53-9efa-12a2f95cf773 2024-09-15T02:51:58.696 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:58.695+0000 7fa66b10f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:58.698 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:58.697+0000 7fa66b10f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:58.699 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:58.698+0000 7fa66b10f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:51:58.699 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:58.698+0000 7fa66b10f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:51:58.700 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:58.698+0000 7fa66b10f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:51:58.700 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:58.698+0000 7fa66b10f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:51:58.738 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:51:58.737+0000 7fa66b10f440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:52:04.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:52:04.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:52:04.762 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:52:04.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:52:04.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:52:05.254 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:52:05.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:52:05.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:52:05.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:52:05.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:52:05.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:52:05.277 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:05.276+0000 7f3ad9b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:05.279 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:05.278+0000 7f3ad9b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:05.280 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:05.279+0000 7f3ad9b1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:05.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:52:05.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:52:05.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:52:05.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:52:05.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:52:05.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:05.605 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:52:05.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:52:05.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:05.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:05.804 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:05.803+0000 7f3ad9b1f440 -1 Falling back to public interface 2024-09-15T02:52:05.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:06.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:06.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:06.964 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:52:06.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:52:06.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:06.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:07.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:08.276 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:08.275+0000 7f3ad9b1f440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:52:08.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:08.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:08.324 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:52:08.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:52:08.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:08.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:08.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:09.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:09.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:09.877 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:52:09.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:52:09.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:09.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:10.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:11.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:11.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:11.265 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:52:11.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:52:11.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:11.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:11.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:12.645 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:52:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:52:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:13.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:14.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:14.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:14.152 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:52:14.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:52:14.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:14.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:14.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:15.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:15.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:15.514 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:52:15.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:52:15.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:15.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:15.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:16.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:16.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:16.989 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:52:16.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:52:16.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:16.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:17.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:18.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:18.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:18.358 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:52:18.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:52:18.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:18.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:18.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:19.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:19.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:19.749 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:52:19.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:52:19.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:19.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:20.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:21.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:21.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:21.362 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:52:21.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:52:21.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:21.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:22.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:23.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:23.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:23.071 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:52:23.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:52:23.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:23.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:24.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:24.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:24.593 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:52:24.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:52:24.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:24.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:25.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:26.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:26.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:26.102 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:52:26.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:52:26.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:26.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:26.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:27.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:27.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:27.850 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:52:27.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:52:27.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:27.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:28.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:29.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:29.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:29.669 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:52:29.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:52:29.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:29.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:30.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:31.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:31.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:31.552 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:52:31.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:52:31.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:31.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:52:32.008 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/188884,v1:127.0.0.1:6803/188884] [v2:127.0.0.1:6804/188884,v1:127.0.0.1:6805/188884] exists,up 4684e24c-c8d5-4d53-9efa-12a2f95cf773 2024-09-15T02:52:32.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:52:32.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:52:32.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:52:32.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:52:32.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:52:32.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:52:32.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:52:32.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:52:32.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0644c013-279f-4625-aad8-ecb2d8d455f1 2024-09-15T02:52:32.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0644c013-279f-4625-aad8-ecb2d8d455f1' 2024-09-15T02:52:32.016 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 0644c013-279f-4625-aad8-ecb2d8d455f1 2024-09-15T02:52:32.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:52:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwS+ZmTWmNARAAKgCwpFFtQ+7BuZlI2GvRIw== 2024-09-15T02:52:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwS+ZmTWmNARAAKgCwpFFtQ+7BuZlI2GvRIw=="}' 2024-09-15T02:52:32.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0644c013-279f-4625-aad8-ecb2d8d455f1 -i td/test-erasure-eio/1/new.json 2024-09-15T02:52:32.510 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:52:32.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:52:32.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwS+ZmTWmNARAAKgCwpFFtQ+7BuZlI2GvRIw== --osd-uuid 0644c013-279f-4625-aad8-ecb2d8d455f1 2024-09-15T02:52:32.546 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:32.545+0000 7f82cce1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:32.548 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:32.547+0000 7f82cce1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:32.549 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:32.548+0000 7f82cce1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:32.550 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:32.548+0000 7f82cce1c440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:52:32.550 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:32.548+0000 7f82cce1c440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:52:32.550 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:32.548+0000 7f82cce1c440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:52:32.626 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:32.625+0000 7f82cce1c440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:52:38.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:52:38.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:52:38.504 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:52:38.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:52:38.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:52:39.017 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:52:39.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:52:39.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:52:39.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:52:39.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:52:39.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:52:39.039 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:39.038+0000 7f2f36a48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:39.041 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:39.040+0000 7f2f36a48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:39.042 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:39.041+0000 7f2f36a48440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:52:39.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:52:39.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:52:39.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:52:39.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:52:39.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:52:39.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:39.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:52:39.365 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:52:39.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:39.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:39.563 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:39.562+0000 7f2f36a48440 -1 Falling back to public interface 2024-09-15T02:52:39.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:40.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:40.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:40.753 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:52:40.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:52:40.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:40.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:41.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:42.066 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:52:42.064+0000 7f2f36a48440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:52:42.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:42.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:42.137 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:52:42.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:52:42.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:42.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:42.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:43.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:43.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:43.834 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:52:43.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:52:43.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:43.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:44.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:45.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:45.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:45.214 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:52:45.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:52:45.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:45.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:45.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:46.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:46.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:46.586 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:52:46.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:52:46.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:46.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:47.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:48.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:48.034 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:52:48.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:52:48.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:48.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:48.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:49.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:49.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:49.396 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:52:49.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:52:49.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:49.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:49.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:50.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:50.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:50.821 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:52:50.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:52:50.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:50.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:51.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:52.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:52.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:52.181 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:52:52.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:52:52.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:52.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:53.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:53.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:53.564 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:52:53.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:52:53.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:53.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:54.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:55.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:55.074 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:52:55.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:52:55.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:55.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:55.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:56.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:56.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:56.938 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:52:56.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:52:56.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:56.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:57.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:52:58.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:52:58.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:52:58.782 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:52:58.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:52:58.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:52:58.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:52:59.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:00.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:00.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:00.648 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:53:00.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:53:00.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:00.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:53:01.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:02.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:02.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:02.602 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:53:02.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:53:02.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:02.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:53:03.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:04.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:04.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:04.729 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:53:04.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:53:04.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:04.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:53:05.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:06.284 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:53:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:53:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:06.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:53:06.697 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/190638,v1:127.0.0.1:6811/190638] [v2:127.0.0.1:6812/190638,v1:127.0.0.1:6813/190638] exists,up 0644c013-279f-4625-aad8-ecb2d8d455f1 2024-09-15T02:53:06.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:53:06.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:53:06.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:53:06.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:53:06.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:53:06.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:53:06.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:53:06.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:53:06.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2b01399-a77c-4256-9535-cdef63326b9b 2024-09-15T02:53:06.704 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 a2b01399-a77c-4256-9535-cdef63326b9b 2024-09-15T02:53:06.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a2b01399-a77c-4256-9535-cdef63326b9b' 2024-09-15T02:53:06.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:53:06.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASTOZmUiqbKhAA/brE3Z6SLWK7IpYsWyDjkw== 2024-09-15T02:53:06.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASTOZmUiqbKhAA/brE3Z6SLWK7IpYsWyDjkw=="}' 2024-09-15T02:53:06.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2b01399-a77c-4256-9535-cdef63326b9b -i td/test-erasure-eio/2/new.json 2024-09-15T02:53:07.199 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:53:07.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:53:07.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASTOZmUiqbKhAA/brE3Z6SLWK7IpYsWyDjkw== --osd-uuid a2b01399-a77c-4256-9535-cdef63326b9b 2024-09-15T02:53:07.236 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:07.233+0000 7fdc12008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:07.236 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:07.235+0000 7fdc12008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:07.237 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:07.236+0000 7fdc12008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:07.238 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:07.236+0000 7fdc12008440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:53:07.238 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:07.236+0000 7fdc12008440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:53:07.238 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:07.236+0000 7fdc12008440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:53:07.289 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:07.287+0000 7fdc12008440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:53:13.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:53:13.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:53:13.070 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:53:13.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:53:13.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:53:13.575 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:53:13.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:53:13.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:53:13.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:53:13.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:53:13.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:53:13.597 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:13.595+0000 7f0ad243c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:13.599 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:13.597+0000 7f0ad243c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:13.600 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:13.598+0000 7f0ad243c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:13.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:53:13.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:53:13.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:53:13.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:53:13.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:53:13.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:13.926 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:53:13.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:53:13.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:13.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:14.115 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:14.113+0000 7f0ad243c440 -1 Falling back to public interface 2024-09-15T02:53:14.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:15.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:15.277 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:53:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:53:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:15.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:16.580 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:16.578+0000 7f0ad243c440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:53:16.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:16.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:16.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:53:16.628 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:53:16.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:16.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:17.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:18.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:18.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:18.023 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:53:18.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:53:18.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:18.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:18.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:19.504 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:53:19.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:53:19.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:19.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:20.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:21.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:21.056 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:53:21.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:53:21.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:21.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:21.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:22.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:22.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:22.539 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:53:22.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:53:22.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:22.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:22.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:23.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:23.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:23.963 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:53:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:53:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:24.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:25.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:25.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:25.332 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:53:25.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:53:25.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:25.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:25.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:26.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:26.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:26.758 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:53:26.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:53:26.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:26.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:27.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:28.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:28.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:28.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:53:28.115 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:53:28.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:28.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:28.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:29.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:29.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:29.606 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:53:29.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:53:29.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:29.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:30.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:31.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:31.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:31.334 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:53:31.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:53:31.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:31.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:32.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:33.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:33.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:33.341 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:53:33.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:53:33.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:33.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:34.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:35.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:35.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:35.251 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:53:35.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:53:35.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:35.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:36.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:37.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:37.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:37.045 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:53:37.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:53:37.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:37.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:38.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:39.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:39.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:39.038 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:53:39.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:53:39.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:39.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:39.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:40.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:40.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:40.880 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:53:40.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:53:40.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:40.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:53:41.489 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/192395,v1:127.0.0.1:6819/192395] [v2:127.0.0.1:6820/192395,v1:127.0.0.1:6821/192395] exists,up a2b01399-a77c-4256-9535-cdef63326b9b 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:53:41.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:53:41.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:53:41.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0113b79-ec9e-4acf-9975-d9c3cf5ab4d1 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b0113b79-ec9e-4acf-9975-d9c3cf5ab4d1' 2024-09-15T02:53:41.496 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 b0113b79-ec9e-4acf-9975-d9c3cf5ab4d1 2024-09-15T02:53:41.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:53:41.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1TOZmAfgrHhAAk7QzYFXspEEeMtff/j30iw== 2024-09-15T02:53:41.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1TOZmAfgrHhAAk7QzYFXspEEeMtff/j30iw=="}' 2024-09-15T02:53:41.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0113b79-ec9e-4acf-9975-d9c3cf5ab4d1 -i td/test-erasure-eio/3/new.json 2024-09-15T02:53:42.131 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:53:42.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:53:42.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1TOZmAfgrHhAAk7QzYFXspEEeMtff/j30iw== --osd-uuid b0113b79-ec9e-4acf-9975-d9c3cf5ab4d1 2024-09-15T02:53:42.167 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:42.166+0000 7f01bbf88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:42.171 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:42.170+0000 7f01bbf88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:42.172 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:42.171+0000 7f01bbf88440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:42.173 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:42.172+0000 7f01bbf88440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:53:42.173 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:42.172+0000 7f01bbf88440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:53:42.173 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:42.172+0000 7f01bbf88440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:53:42.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:42.254+0000 7f01bbf88440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:53:48.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:53:48.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:53:48.429 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:53:48.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:53:48.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:53:48.916 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:53:48.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:53:48.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:53:48.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:53:48.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:53:48.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:53:48.939 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:48.938+0000 7f1db5e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:48.941 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:48.940+0000 7f1db5e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:48.942 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:48.941+0000 7f1db5e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:53:49.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:53:49.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:53:49.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:53:49.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:53:49.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:53:49.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:49.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:53:49.309 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:53:49.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:49.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:49.457 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:49.456+0000 7f1db5e09440 -1 Falling back to public interface 2024-09-15T02:53:49.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:50.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:50.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:50.663 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:53:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:53:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:50.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:51.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:51.934 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:53:51.933+0000 7f1db5e09440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:53:52.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:52.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:52.012 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:53:52.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:53:52.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:52.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:52.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:53.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:53.492 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:53:53.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:53:53.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:53.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:54.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:55.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:55.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:55.177 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:53:55.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:53:55.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:55.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:55.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:56.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:56.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:56.744 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:53:56.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:53:56.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:56.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:57.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:58.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:58.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:58.115 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:53:58.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:53:58.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:58.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:53:58.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:53:59.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:53:59.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:53:59.513 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:53:59.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:53:59.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:53:59.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:00.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:01.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:01.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:01.003 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:54:01.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:54:01.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:01.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:01.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:02.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:02.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:02.460 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:54:02.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:54:02.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:02.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:02.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:03.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:03.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:03.899 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:54:03.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:54:03.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:03.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:04.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:05.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:05.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:05.288 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:54:05.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:54:05.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:05.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:05.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:06.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:06.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:06.954 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:54:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:54:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:06.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:07.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:08.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:08.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:08.992 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:54:08.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:54:08.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:08.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:09.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:10.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:10.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:10.951 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:54:10.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:54:10.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:10.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:11.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:12.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:12.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:12.852 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:54:12.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:54:12.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:12.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:14.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:15.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:15.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:15.150 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:54:15.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:54:15.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:15.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:15.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:16.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:16.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:16.539 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:54:16.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:54:16.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:16.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:54:17.010 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/194149,v1:127.0.0.1:6827/194149] [v2:127.0.0.1:6828/194149,v1:127.0.0.1:6829/194149] exists,up b0113b79-ec9e-4acf-9975-d9c3cf5ab4d1 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:54:17.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:54:17.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:54:17.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-15T02:54:17.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:54:17.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fee17131-e0c9-43c2-8778-8d99729f7d4e 2024-09-15T02:54:17.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 fee17131-e0c9-43c2-8778-8d99729f7d4e' 2024-09-15T02:54:17.017 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 fee17131-e0c9-43c2-8778-8d99729f7d4e 2024-09-15T02:54:17.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:54:17.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZTOZmusKdARAAgDfIPM3qJXTDokOJdSi8NQ== 2024-09-15T02:54:17.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZTOZmusKdARAAgDfIPM3qJXTDokOJdSi8NQ=="}' 2024-09-15T02:54:17.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fee17131-e0c9-43c2-8778-8d99729f7d4e -i td/test-erasure-eio/4/new.json 2024-09-15T02:54:17.929 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:54:17.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-15T02:54:17.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZTOZmusKdARAAgDfIPM3qJXTDokOJdSi8NQ== --osd-uuid fee17131-e0c9-43c2-8778-8d99729f7d4e 2024-09-15T02:54:17.965 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:17.965+0000 7fa57080b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:17.993 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:17.992+0000 7fa57080b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:17.994 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:17.993+0000 7fa57080b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:17.994 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:17.994+0000 7fa57080b440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:54:17.994 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:17.994+0000 7fa57080b440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:54:17.995 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:17.994+0000 7fa57080b440 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-15T02:54:18.179 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:18.178+0000 7fa57080b440 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-15T02:54:24.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-15T02:54:24.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:54:24.359 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-15T02:54:24.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T02:54:24.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:54:24.867 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-15T02:54:24.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T02:54:24.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:54:24.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:54:24.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:54:24.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:54:24.890 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:24.889+0000 7f8568231440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:24.891 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:24.891+0000 7f8568231440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:24.893 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:24.892+0000 7f8568231440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:54:25.227 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:54:25.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:25.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:25.415 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:25.414+0000 7f8568231440 -1 Falling back to public interface 2024-09-15T02:54:25.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:26.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:26.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:26.597 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:54:26.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:54:26.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:26.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:26.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:27.902 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:27.901+0000 7f8568231440 -1 osd.4 0 log_to_monitors true 2024-09-15T02:54:27.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:27.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:27.998 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:54:27.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:54:27.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:27.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:28.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:29.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:29.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:29.451 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:54:29.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:54:29.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:29.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:29.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:30.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:30.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:30.860 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:54:30.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:54:30.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:30.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:31.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:32.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:32.496 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:54:32.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:54:32.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:32.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:33.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:34.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:34.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:34.109 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:54:34.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:54:34.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:34.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:34.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:35.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:35.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:35.469 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:54:35.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:54:35.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:35.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:36.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:36.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:36.897 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:54:36.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:54:36.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:36.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:37.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:38.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:38.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:38.329 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:54:38.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:54:38.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:38.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:38.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:39.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:39.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:39.867 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:54:39.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:54:39.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:39.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:40.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:41.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:41.273 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:54:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:54:41.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:41.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:42.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:43.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:43.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:43.020 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:54:43.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:54:43.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:43.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:44.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:45.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:45.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:45.176 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:54:45.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:54:45.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:45.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:45.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:46.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:46.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:46.899 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:54:46.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:54:46.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:46.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:47.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:48.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:48.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:48.726 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:54:48.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:54:48.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:48.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:49.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:50.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:50.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:50.570 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:54:50.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:54:50.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:50.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:51.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:54:52.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:54:52.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:54:52.418 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:54:52.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:54:52.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:54:52.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T02:54:52.899 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/195903,v1:127.0.0.1:6835/195903] [v2:127.0.0.1:6836/195903,v1:127.0.0.1:6837/195903] exists,up fee17131-e0c9-43c2-8778-8d99729f7d4e 2024-09-15T02:54:52.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:54:52.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:54:52.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:54:52.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:54:52.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-15T02:54:52.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:54:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:54:52.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:54:52.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff3884f9-982e-48dd-8222-fffd5dc0e8ff 2024-09-15T02:54:52.907 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 ff3884f9-982e-48dd-8222-fffd5dc0e8ff 2024-09-15T02:54:52.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ff3884f9-982e-48dd-8222-fffd5dc0e8ff' 2024-09-15T02:54:52.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:54:52.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8TOZmTwGqNhAAJdqKiHPA3dShtGeVt8eLvg== 2024-09-15T02:54:52.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8TOZmTwGqNhAAJdqKiHPA3dShtGeVt8eLvg=="}' 2024-09-15T02:54:52.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff3884f9-982e-48dd-8222-fffd5dc0e8ff -i td/test-erasure-eio/5/new.json 2024-09-15T02:54:54.096 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:54:54.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-15T02:54:54.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8TOZmTwGqNhAAJdqKiHPA3dShtGeVt8eLvg== --osd-uuid ff3884f9-982e-48dd-8222-fffd5dc0e8ff 2024-09-15T02:54:54.132 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:54.131+0000 7f2ae6d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:54.134 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:54.133+0000 7f2ae6d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:54.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:54.134+0000 7f2ae6d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:54:54.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:54.134+0000 7f2ae6d23440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:54:54.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:54.134+0000 7f2ae6d23440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:54:54.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:54.134+0000 7f2ae6d23440 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-15T02:54:54.214 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:54:54.213+0000 7f2ae6d23440 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-15T02:55:00.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-15T02:55:00.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:55:00.792 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-15T02:55:00.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T02:55:00.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:55:01.274 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-15T02:55:01.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T02:55:01.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:55:01.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:55:01.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:55:01.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:55:01.296 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:01.294+0000 7f3039329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:01.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:01.296+0000 7f3039329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:01.299 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:01.297+0000 7f3039329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:01.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T02:55:01.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:55:01.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T02:55:01.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:55:01.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:55:01.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:01.627 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:55:01.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:55:01.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:01.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:01.817 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:01.815+0000 7f3039329440 -1 Falling back to public interface 2024-09-15T02:55:01.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:02.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:02.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:02.984 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:55:02.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:55:02.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:02.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:03.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:04.308 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:04.306+0000 7f3039329440 -1 osd.5 0 log_to_monitors true 2024-09-15T02:55:04.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:04.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:04.358 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:55:04.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:55:04.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:04.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:04.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:05.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:05.873 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:55:05.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:05.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:55:05.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:05.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:06.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:07.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:07.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:07.428 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:55:07.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:55:07.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:07.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:07.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:08.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:08.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:08.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:55:08.816 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:55:08.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:08.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:09.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:10.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:10.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:10.241 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:55:10.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:55:10.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:10.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:10.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:11.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:11.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:11.601 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:55:11.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:55:11.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:11.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:12.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:13.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:13.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:13.107 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:55:13.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:55:13.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:13.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:13.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:14.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:14.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:14.527 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:55:14.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:55:14.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:14.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:14.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:15.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:15.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:15.917 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:55:15.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:55:15.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:15.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:16.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:17.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:17.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:17.398 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:55:17.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:55:17.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:17.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:18.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:19.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:19.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:19.144 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:55:19.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:55:19.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:19.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:20.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:21.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:21.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:21.211 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:55:21.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:55:21.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:21.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:21.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:22.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:22.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:22.770 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:55:22.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:55:22.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:22.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:23.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:24.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:24.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:24.598 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:55:24.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:55:24.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:24.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:25.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:26.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:26.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:26.131 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:55:26.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:55:26.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:26.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:26.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:27.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:27.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:27.812 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:55:27.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:55:27.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:27.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:28.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:29.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:29.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:29.508 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T02:55:29.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T02:55:29.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:29.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T02:55:29.925 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/197658,v1:127.0.0.1:6843/197658] [v2:127.0.0.1:6844/197658,v1:127.0.0.1:6845/197658] exists,up ff3884f9-982e-48dd-8222-fffd5dc0e8ff 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-15T02:55:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:55:29.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:55:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:55:29.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-15T02:55:29.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:55:29.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=653dd940-5269-477a-86d2-00aa62e6a357 2024-09-15T02:55:29.932 INFO:tasks.workunit.client.0.smithi130.stdout:add osd6 653dd940-5269-477a-86d2-00aa62e6a357 2024-09-15T02:55:29.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 653dd940-5269-477a-86d2-00aa62e6a357' 2024-09-15T02:55:29.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:55:29.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChTOZm0lwvOBAAy0y9kJyj5Zo+EO5OYD7JCw== 2024-09-15T02:55:29.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChTOZm0lwvOBAAy0y9kJyj5Zo+EO5OYD7JCw=="}' 2024-09-15T02:55:29.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 653dd940-5269-477a-86d2-00aa62e6a357 -i td/test-erasure-eio/6/new.json 2024-09-15T02:55:30.701 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:55:30.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-15T02:55:30.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChTOZm0lwvOBAAy0y9kJyj5Zo+EO5OYD7JCw== --osd-uuid 653dd940-5269-477a-86d2-00aa62e6a357 2024-09-15T02:55:30.737 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:30.736+0000 7fdb9a723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:30.752 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:30.751+0000 7fdb9a723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:30.752 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:30.752+0000 7fdb9a723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:30.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:30.752+0000 7fdb9a723440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:55:30.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:30.752+0000 7fdb9a723440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:55:30.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:30.752+0000 7fdb9a723440 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-15T02:55:30.844 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:30.843+0000 7fdb9a723440 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-15T02:55:37.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-15T02:55:37.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:55:37.226 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd6 key to auth repository 2024-09-15T02:55:37.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-15T02:55:37.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:55:37.727 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.6 2024-09-15T02:55:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-15T02:55:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:55:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:55:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:55:37.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:55:37.750 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:37.749+0000 7f9eb3a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:37.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:37.752+0000 7f9eb3a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:37.753 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:37.753+0000 7f9eb3a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:55:38.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-15T02:55:38.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:55:38.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-15T02:55:38.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:55:38.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:55:38.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:38.079 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:55:38.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:55:38.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:38.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:38.277 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:38.276+0000 7f9eb3a75440 -1 Falling back to public interface 2024-09-15T02:55:38.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:39.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:39.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:39.440 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:55:39.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:55:39.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:39.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:39.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:40.797 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:55:40.796+0000 7f9eb3a75440 -1 osd.6 0 log_to_monitors true 2024-09-15T02:55:40.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:40.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:40.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:55:40.798 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:55:40.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:40.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:41.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:42.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:42.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:42.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:55:42.155 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:55:42.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:42.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:42.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:43.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:43.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:43.882 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:55:43.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:55:43.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:43.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:44.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:45.273 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:55:45.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:55:45.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:45.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:45.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:46.723 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:55:46.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:55:46.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:46.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:47.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:48.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:48.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:48.104 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:55:48.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:55:48.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:48.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:48.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:49.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:49.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:49.628 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:55:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:55:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:50.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:51.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:51.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:51.009 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:55:51.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:55:51.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:51.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:51.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:52.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:52.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:52.493 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:55:52.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:55:52.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:52.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:52.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:53.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:53.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:53.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:55:53.883 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:55:53.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:53.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:54.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:55.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:55.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:55.402 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:55:55.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:55:55.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:55.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:56.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:57.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:57.533 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:55:57.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:57.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:55:57.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:57.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:58.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:55:59.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:55:59.016 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:55:59.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:55:59.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:55:59.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:55:59.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:55:59.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:00.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:00.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:00.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:56:00.689 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:56:00.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:00.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:56:01.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:02.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:02.805 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:56:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:56:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:02.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:56:03.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:04.888 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:56:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:56:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:04.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-15T02:56:05.341 INFO:tasks.workunit.client.0.smithi130.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/199437,v1:127.0.0.1:6851/199437] [v2:127.0.0.1:6852/199437,v1:127.0.0.1:6853/199437] exists,up 653dd940-5269-477a-86d2-00aa62e6a357 2024-09-15T02:56:05.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:56:05.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:56:05.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:56:05.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:56:05.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:56:05.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:56:05.343 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:05.343 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:05.343 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:05.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:56:05.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:56:05.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:56:05.416 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:56:05.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:56:05.428 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:52:05.809+0000 7f3ad9b1f440 0 load: jerasure load: lrc 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:56:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-15T02:56:06.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:56:06.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:56:07.773 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:56:07.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:56:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:56:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:56:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:56:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:56:08.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:56:08.787 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:56:08.787 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:56:08.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:56:08.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:56:08.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:56:08.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:56:08.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:56:08.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:56:08.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:56:08.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:56:08.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:56:09.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:56:09.254 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:56:09.254 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:56:09.254 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:56:09.254 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:56:09.254 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:56:09.254 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:56:09.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:56:09.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:09.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:56:09.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803825 2024-09-15T02:56:09.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803825 2024-09-15T02:56:09.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825' 2024-09-15T02:56:09.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:09.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:56:09.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574888 2024-09-15T02:56:09.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574888 2024-09-15T02:56:09.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574888' 2024-09-15T02:56:09.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:09.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:56:09.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378658 2024-09-15T02:56:09.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378658 2024-09-15T02:56:09.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574888 2-81604378658' 2024-09-15T02:56:09.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:09.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:56:09.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084315 2024-09-15T02:56:09.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084315 2024-09-15T02:56:09.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574888 2-81604378658 3-120259084315' 2024-09-15T02:56:09.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:09.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:56:09.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724566 2024-09-15T02:56:09.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724566 2024-09-15T02:56:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574888 2-81604378658 3-120259084315 4-167503724566' 2024-09-15T02:56:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:09.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:56:10.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364817 2024-09-15T02:56:10.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364817 2024-09-15T02:56:10.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574888 2-81604378658 3-120259084315 4-167503724566 5-214748364817' 2024-09-15T02:56:10.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:10.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:56:10.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070475 2024-09-15T02:56:10.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070475 2024-09-15T02:56:10.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803825 1-55834574888 2-81604378658 3-120259084315 4-167503724566 5-214748364817 6-253403070475' 2024-09-15T02:56:10.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:10.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803825 2024-09-15T02:56:10.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:10.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:56:10.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803825 2024-09-15T02:56:10.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:10.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803825 2024-09-15T02:56:10.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-09-15T02:56:10.132 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803825 2024-09-15T02:56:10.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:56:10.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803823 -lt 25769803825 2024-09-15T02:56:10.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:56:11.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:56:11.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:56:11.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803825 -lt 25769803825 2024-09-15T02:56:11.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:11.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574888 2024-09-15T02:56:11.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:11.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:56:11.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574888 2024-09-15T02:56:11.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574888 2024-09-15T02:56:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574888' 2024-09-15T02:56:11.836 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574888 2024-09-15T02:56:11.836 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:56:12.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574888 -lt 55834574888 2024-09-15T02:56:12.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:12.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378658 2024-09-15T02:56:12.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:12.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:56:12.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378658 2024-09-15T02:56:12.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:12.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378658 2024-09-15T02:56:12.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378658' 2024-09-15T02:56:12.192 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378658 2024-09-15T02:56:12.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:56:12.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378658 -lt 81604378658 2024-09-15T02:56:12.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:12.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084315 2024-09-15T02:56:12.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:12.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:56:12.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084315 2024-09-15T02:56:12.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:12.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084315 2024-09-15T02:56:12.561 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084315 2024-09-15T02:56:12.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084315' 2024-09-15T02:56:12.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:56:12.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084316 -lt 120259084315 2024-09-15T02:56:12.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:12.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724566 2024-09-15T02:56:12.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:12.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:56:12.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724566 2024-09-15T02:56:12.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:12.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724566 2024-09-15T02:56:12.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724566' 2024-09-15T02:56:12.913 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 167503724566 2024-09-15T02:56:12.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:56:13.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724567 -lt 167503724566 2024-09-15T02:56:13.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:13.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-214748364817 2024-09-15T02:56:13.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:13.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:56:13.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-214748364817 2024-09-15T02:56:13.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:13.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364817 2024-09-15T02:56:13.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 214748364817' 2024-09-15T02:56:13.271 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 214748364817 2024-09-15T02:56:13.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:56:13.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364817 -lt 214748364817 2024-09-15T02:56:13.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:13.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-253403070475 2024-09-15T02:56:13.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:13.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-15T02:56:13.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-253403070475 2024-09-15T02:56:13.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:13.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070475 2024-09-15T02:56:13.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 253403070475' 2024-09-15T02:56:13.616 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.6 seq 253403070475 2024-09-15T02:56:13.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-15T02:56:13.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070476 -lt 253403070475 2024-09-15T02:56:13.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:56:13.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:56:13.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:56:14.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:56:14.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:56:14.795 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:56:14.795 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:56:14.795 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:56:15.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:56:15.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:56:15.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:56:15.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:56:15.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:56:15.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:56:15.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-15T02:56:15.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T02:56:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T02:56:15.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T02:56:15.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-15T02:56:15.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:56:15.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:56:15.426 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:56:15.426 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:56:15.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:56:15.823 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:56:15.823 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:56:15.823 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:56:15.823 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:56:15.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-15T02:56:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-09-15T02:56:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T02:56:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T02:56:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T02:56:15.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T02:56:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T02:56:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T02:56:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T02:56:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T02:56:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T02:56:15.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T02:56:15.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T02:56:15.826 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T02:56:15.826 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:56:15.826 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:56:15.826 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:15.826 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:15.826 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:15.827 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:56:15.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T02:56:15.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T02:56:15.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T02:56:15.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T02:56:15.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T02:56:15.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-15T02:56:15.985 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-09-15T02:56:15.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:56:15.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T02:56:15.989 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-15T02:56:15.989 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:56:16.335 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:56:16.335 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:56:16.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:56:16.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:56:16.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2024-09-15T02:56:17.576 INFO:tasks.workunit.client.0.smithi130.stderr:osd.6 is already down. 2024-09-15T02:56:17.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2024-09-15T02:56:18.076 INFO:tasks.workunit.client.0.smithi130.stderr:osd.6 is already out. 2024-09-15T02:56:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2024-09-15T02:56:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:56:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:56:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:56:18.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:56:18.089 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:56:18.089 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:56:18.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:56:18.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:56:18.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:56:18.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:56:18.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:56:18.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:56:18.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:56:18.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:56:18.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:56:18.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:5 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:6' 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:18.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:56:18.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803829 2024-09-15T02:56:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803829 2024-09-15T02:56:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829' 2024-09-15T02:56:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:18.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:56:19.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574892 2024-09-15T02:56:19.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574892 2024-09-15T02:56:19.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-55834574892' 2024-09-15T02:56:19.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:19.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:56:19.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378662 2024-09-15T02:56:19.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378662 2024-09-15T02:56:19.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-55834574892 2-81604378662' 2024-09-15T02:56:19.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:19.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:56:19.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084319 2024-09-15T02:56:19.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084319 2024-09-15T02:56:19.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-55834574892 2-81604378662 3-120259084319' 2024-09-15T02:56:19.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:19.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T02:56:19.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724570 2024-09-15T02:56:19.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724570 2024-09-15T02:56:19.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-55834574892 2-81604378662 3-120259084319 4-167503724570' 2024-09-15T02:56:19.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:19.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T02:56:19.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364820 2024-09-15T02:56:19.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364820 2024-09-15T02:56:19.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-55834574892 2-81604378662 3-120259084319 4-167503724570 5-214748364820' 2024-09-15T02:56:19.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:56:19.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-15T02:56:19.720 INFO:tasks.workunit.client.0.smithi130.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-15T02:56:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T02:56:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T02:56:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T02:56:19.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:19.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803829 2024-09-15T02:56:19.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:19.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:56:19.724 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803829 2024-09-15T02:56:19.724 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803829 2024-09-15T02:56:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-09-15T02:56:19.725 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803829 2024-09-15T02:56:19.725 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:56:20.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803827 -lt 25769803829 2024-09-15T02:56:20.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:56:21.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:56:21.076 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:56:21.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803829 -lt 25769803829 2024-09-15T02:56:21.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:21.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574892 2024-09-15T02:56:21.433 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:21.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:56:21.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574892 2024-09-15T02:56:21.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:21.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574892 2024-09-15T02:56:21.436 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574892 2024-09-15T02:56:21.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574892' 2024-09-15T02:56:21.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:56:21.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574892 -lt 55834574892 2024-09-15T02:56:21.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:21.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378662 2024-09-15T02:56:21.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:21.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:56:21.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378662 2024-09-15T02:56:21.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:21.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378662 2024-09-15T02:56:21.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-09-15T02:56:21.787 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378662 2024-09-15T02:56:21.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:56:22.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378662 -lt 81604378662 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084319 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084319 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084319 2024-09-15T02:56:22.143 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084319 2024-09-15T02:56:22.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084319' 2024-09-15T02:56:22.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:56:22.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084319 -lt 120259084319 2024-09-15T02:56:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:22.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724570 2024-09-15T02:56:22.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:22.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T02:56:22.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724570 2024-09-15T02:56:22.489 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:22.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724570 2024-09-15T02:56:22.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724570' 2024-09-15T02:56:22.491 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 167503724570 2024-09-15T02:56:22.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T02:56:22.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724571 -lt 167503724570 2024-09-15T02:56:22.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:56:22.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-214748364820 2024-09-15T02:56:22.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:56:22.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T02:56:22.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-214748364820 2024-09-15T02:56:22.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:56:22.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364820 2024-09-15T02:56:22.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 214748364820' 2024-09-15T02:56:22.846 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 214748364820 2024-09-15T02:56:22.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T02:56:23.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364821 -lt 214748364820 2024-09-15T02:56:23.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:56:23.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:56:23.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:56:23.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:56:24.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:56:24.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:56:24.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:56:24.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:56:24.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:56:24.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:56:24.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:56:24.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-15T02:56:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:56:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:56:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-15T02:56:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:56:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:56:24.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-15T02:56:24.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:56:24.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:56:24.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-09-15T02:56:24.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:56:24.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T02:56:25.052 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T02:56:25.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T02:56:25.634 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:56:25.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:56:25.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:56:25.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:56:25.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:56:25.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:56:25.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:56:25.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:56:25.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:56:25.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:25.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:56:25.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:56:25.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:25.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:56:25.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:56:25.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:56:25.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:56:25.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:25.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:25.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:56:25.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:56:25.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:56:25.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:56:25.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:56:25.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T02:56:25.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T02:56:25.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T02:56:25.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T02:56:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T02:56:25.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T02:56:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:25.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T02:56:25.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T02:56:25.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:25.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T02:56:25.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T02:56:25.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T02:56:25.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T02:56:25.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:25.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:25.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T02:56:25.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T02:56:25.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T02:56:25.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T02:56:25.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T02:56:25.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:25.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:25.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T02:56:25.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T02:56:25.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T02:56:26.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T02:56:26.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:56:26.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:56:26.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:56:26.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:26.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:26.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:56:26.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T02:56:26.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T02:56:26.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T02:56:26.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:56:26.395 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:26.396 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:56:26.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:56:26.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T02:56:26.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T02:56:26.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T02:56:26.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T02:56:26.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:56:26.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T02:56:26.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T02:56:26.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T02:56:26.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T02:56:26.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T02:56:26.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T02:56:26.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T02:56:26.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T02:56:26.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T02:56:27.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T02:56:27.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:56:27.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:56:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:56:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:27.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:56:27.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:56:27.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T02:56:27.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T02:56:27.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T02:56:27.714 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T02:56:27.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:56:28.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T02:56:28.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T02:56:28.726 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T02:56:28.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:56:28.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:28.726 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:28.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T02:56:28.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T02:56:28.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T02:56:28.795 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:56:28.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T02:56:28.804 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:56:26.187+0000 7fcfcda37900 0 load: jerasure load: lrc 2024-09-15T02:56:28.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-09-15T02:56:28.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-09-15T02:56:28.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-09-15T02:56:28.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T02:56:28.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T02:56:28.805 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T02:56:28.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T02:56:28.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:56:28.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:56:28.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T02:56:28.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T02:56:28.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:56:28.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:56:28.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:56:28.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:56:28.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T02:56:28.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:56:28.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ede0355d-c99a-43f4-9408-e00650e50467 2024-09-15T02:56:28.814 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 ede0355d-c99a-43f4-9408-e00650e50467 2024-09-15T02:56:28.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ede0355d-c99a-43f4-9408-e00650e50467' 2024-09-15T02:56:28.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:56:28.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcTOZm1lsaMRAAPDTaysJ3RxO1ofUq8VJ3UQ== 2024-09-15T02:56:28.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcTOZm1lsaMRAAPDTaysJ3RxO1ofUq8VJ3UQ=="}' 2024-09-15T02:56:28.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ede0355d-c99a-43f4-9408-e00650e50467 -i td/test-erasure-eio/0/new.json 2024-09-15T02:56:29.093 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:56:29.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T02:56:29.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcTOZm1lsaMRAAPDTaysJ3RxO1ofUq8VJ3UQ== --osd-uuid ede0355d-c99a-43f4-9408-e00650e50467 2024-09-15T02:56:29.127 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:29.126+0000 7fc99980d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:29.138 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:29.138+0000 7fc99980d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:29.139 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:29.139+0000 7fc99980d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:29.140 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:29.139+0000 7fc99980d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:56:29.140 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:29.139+0000 7fc99980d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:56:29.140 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:29.139+0000 7fc99980d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T02:56:29.178 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:29.177+0000 7fc99980d440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T02:56:35.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T02:56:35.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:56:35.211 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T02:56:35.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T02:56:35.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:56:35.704 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T02:56:35.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T02:56:35.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:56:35.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:56:35.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:56:35.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:56:35.727 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:35.726+0000 7f147520f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:35.729 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:35.729+0000 7f147520f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:35.731 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:35.730+0000 7f147520f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:56:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T02:56:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:56:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T02:56:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:56:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:56:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:56:36.057 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:56:36.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:36.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:36.252 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:36.251+0000 7f147520f440 -1 Falling back to public interface 2024-09-15T02:56:36.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:37.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:37.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:37.411 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:56:37.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:56:37.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:37.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:37.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:38.739 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:56:38.738+0000 7f147520f440 -1 osd.0 0 log_to_monitors true 2024-09-15T02:56:38.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:38.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:38.768 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:56:38.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:56:38.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:38.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:39.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:40.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:40.298 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:56:40.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:40.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:56:40.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:40.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:40.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:41.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:41.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:41.720 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:56:41.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:56:41.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:41.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:42.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:43.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:43.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:43.104 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:56:43.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:56:43.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:43.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:43.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:44.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:44.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:44.548 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:56:44.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:56:44.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:44.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:44.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:45.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:45.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:45.946 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:56:45.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:56:45.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:45.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:46.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:47.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:47.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:47.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:56:47.468 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:56:47.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:47.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:47.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:48.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:48.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:48.853 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:56:48.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:56:48.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:48.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:49.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:50.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:50.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:50.268 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:56:50.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:56:50.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:50.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:51.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:52.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:52.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:52.192 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:56:52.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:56:52.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:52.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:52.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:53.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:53.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:53.799 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:56:53.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:56:53.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:53.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:54.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:55.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:55.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:55.563 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:56:55.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:56:55.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:55.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:56.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:57.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:57.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:57.215 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:56:57.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:56:57.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:57.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:56:59.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:56:59.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:56:59.261 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:56:59.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:56:59.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:56:59.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:56:59.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:00.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:00.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:00.792 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:57:00.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:57:00.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:00.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:57:01.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:02.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:02.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:02.165 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:57:02.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:57:02.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:02.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T02:57:02.542 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/202825,v1:127.0.0.1:6803/202825] [v2:127.0.0.1:6804/202825,v1:127.0.0.1:6805/202825] exists,up ede0355d-c99a-43f4-9408-e00650e50467 2024-09-15T02:57:02.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:57:02.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:57:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:57:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:57:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T02:57:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:57:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:57:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T02:57:02.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:57:02.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:57:02.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:57:02.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:57:02.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:57:02.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:57:02.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:57:02.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:57:02.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:57:02.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:57:02.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:57:02.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:57:02.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:57:02.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:57:02.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:57:02.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:02.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:57:02.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:57:02.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:57:02.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:57:02.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:57:02.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:57:02.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:57:02.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:57:02.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:57:02.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14174469-0503-4543-a4e2-260af417188e 2024-09-15T02:57:02.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 14174469-0503-4543-a4e2-260af417188e' 2024-09-15T02:57:02.556 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 14174469-0503-4543-a4e2-260af417188e 2024-09-15T02:57:02.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:57:02.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+TOZmLJhcIRAAwMJjfR/+AmJmbi4WWo/KwA== 2024-09-15T02:57:02.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+TOZmLJhcIRAAwMJjfR/+AmJmbi4WWo/KwA=="}' 2024-09-15T02:57:02.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14174469-0503-4543-a4e2-260af417188e -i td/test-erasure-eio/1/new.json 2024-09-15T02:57:03.029 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:57:03.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T02:57:03.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+TOZmLJhcIRAAwMJjfR/+AmJmbi4WWo/KwA== --osd-uuid 14174469-0503-4543-a4e2-260af417188e 2024-09-15T02:57:03.065 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:03.064+0000 7f5bb1f31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:03.067 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:03.066+0000 7f5bb1f31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:03.068 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:03.067+0000 7f5bb1f31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:03.068 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:03.067+0000 7f5bb1f31440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:57:03.068 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:03.067+0000 7f5bb1f31440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:57:03.069 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:03.067+0000 7f5bb1f31440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T02:57:03.153 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:03.152+0000 7f5bb1f31440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T02:57:08.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T02:57:08.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:57:08.833 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T02:57:08.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T02:57:08.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:57:09.317 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T02:57:09.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T02:57:09.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:57:09.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:57:09.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:57:09.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:57:09.340 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:09.339+0000 7ffaa8c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:09.350 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:09.349+0000 7ffaa8c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:09.351 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:09.350+0000 7ffaa8c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:09.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T02:57:09.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:57:09.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:57:09.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:57:09.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:57:09.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:09.731 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:57:09.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:57:09.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:09.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:09.883 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:09.882+0000 7ffaa8c09440 -1 Falling back to public interface 2024-09-15T02:57:10.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:11.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:11.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:11.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:57:11.080 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:57:11.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:11.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:11.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:12.388 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:12.387+0000 7ffaa8c09440 -1 osd.1 0 log_to_monitors true 2024-09-15T02:57:12.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:12.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:12.433 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:57:12.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:57:12.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:12.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:13.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:14.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:14.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:14.067 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:57:14.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:57:14.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:14.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:14.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:15.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:15.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:15.462 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:57:15.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:57:15.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:15.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:15.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:16.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:16.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:16.921 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:57:16.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:57:16.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:16.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:17.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:18.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:18.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:18.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:57:18.384 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:57:18.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:18.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:18.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:19.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:19.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:19.749 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:57:19.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:57:19.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:19.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:20.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:21.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:21.173 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:57:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:57:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:21.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:22.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:22.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:22.575 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:57:22.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:57:22.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:22.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:22.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:23.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:23.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:23.977 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:57:23.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:57:23.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:23.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:24.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:25.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:25.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:25.446 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:57:25.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:57:25.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:25.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:26.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:27.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:27.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:27.403 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:57:27.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:57:27.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:27.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:28.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:29.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:29.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:29.289 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:57:29.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:57:29.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:29.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:30.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:31.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:31.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:31.314 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:57:31.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:57:31.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:31.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:32.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:33.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:33.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:33.403 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:57:33.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:57:33.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:33.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:34.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:35.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:35.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:35.288 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:57:35.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:57:35.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:35.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:36.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:37.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:37.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:37.009 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:57:37.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:57:37.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:37.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:57:37.366 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/204579,v1:127.0.0.1:6811/204579] [v2:127.0.0.1:6812/204579,v1:127.0.0.1:6813/204579] exists,up 14174469-0503-4543-a4e2-260af417188e 2024-09-15T02:57:37.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:57:37.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:57:37.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:57:37.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:57:37.375 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:57:37.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7a0f196-efba-4438-828b-999ac89ae58d 2024-09-15T02:57:37.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b7a0f196-efba-4438-828b-999ac89ae58d' 2024-09-15T02:57:37.378 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 b7a0f196-efba-4438-828b-999ac89ae58d 2024-09-15T02:57:37.378 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:57:37.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhTeZm/G3hFhAANk0hn3sGD1Qk6dYxxPKYkw== 2024-09-15T02:57:37.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhTeZm/G3hFhAANk0hn3sGD1Qk6dYxxPKYkw=="}' 2024-09-15T02:57:37.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7a0f196-efba-4438-828b-999ac89ae58d -i td/test-erasure-eio/2/new.json 2024-09-15T02:57:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:57:37.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T02:57:37.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhTeZm/G3hFhAANk0hn3sGD1Qk6dYxxPKYkw== --osd-uuid b7a0f196-efba-4438-828b-999ac89ae58d 2024-09-15T02:57:37.934 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:37.933+0000 7f116b430440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:37.935+0000 7f116b430440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:37.936 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:37.936+0000 7f116b430440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:37.936+0000 7f116b430440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:57:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:37.936+0000 7f116b430440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:57:37.937 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:37.936+0000 7f116b430440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T02:57:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:38.012+0000 7f116b430440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T02:57:43.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T02:57:43.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:57:43.986 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T02:57:43.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T02:57:43.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:57:44.472 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T02:57:44.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T02:57:44.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:57:44.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:57:44.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:57:44.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:57:44.494 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:44.494+0000 7f4bddf80440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:44.496 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:44.496+0000 7f4bddf80440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:44.497 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:44.497+0000 7f4bddf80440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:57:44.834 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:57:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:45.023 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:45.022+0000 7f4bddf80440 -1 Falling back to public interface 2024-09-15T02:57:45.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:46.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:46.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:46.209 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:57:46.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:57:46.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:46.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:46.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:47.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:47.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:47.604 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:57:47.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:57:47.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:47.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:47.663 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:57:47.661+0000 7f4bddf80440 -1 osd.2 0 log_to_monitors true 2024-09-15T02:57:48.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:49.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:49.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:49.126 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:57:49.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:57:49.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:49.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:49.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:50.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:50.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:50.595 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:57:50.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:57:50.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:50.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:50.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:51.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:51.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:51.957 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:57:51.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:57:51.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:51.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:52.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:53.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:53.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:53.355 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:57:53.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:57:53.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:53.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:53.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:54.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:54.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:54.923 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:57:54.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:57:54.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:54.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:55.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:56.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:56.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:56.330 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:57:56.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:57:56.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:56.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:56.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:57.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:57.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:57.768 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:57:57.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:57:57.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:57.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:58.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:57:59.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:57:59.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:57:59.197 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:57:59.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:57:59.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:57:59.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:57:59.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:00.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:00.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:00.681 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:58:00.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:58:00.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:00.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:58:01.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:02.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:02.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:02.201 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:58:02.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:58:02.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:02.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:58:03.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:04.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:04.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:04.054 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:58:04.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:58:04.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:04.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:58:04.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:05.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:05.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:05.903 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:58:05.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:58:05.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:05.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:58:06.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:07.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:07.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:07.911 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:58:07.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:58:07.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:07.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:58:08.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:09.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:58:09.435 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:58:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:09.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:58:10.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:11.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:11.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:11.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:58:11.262 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:58:11.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:11.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T02:58:11.869 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/206333,v1:127.0.0.1:6819/206333] [v2:127.0.0.1:6820/206333,v1:127.0.0.1:6821/206333] exists,up b7a0f196-efba-4438-828b-999ac89ae58d 2024-09-15T02:58:11.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:58:11.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:58:11.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:58:11.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:58:11.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:58:11.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T02:58:11.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T02:58:11.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2df72312-329d-42b8-a241-324d5238dbb0' 2024-09-15T02:58:11.878 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T02:58:11.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T02:58:11.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDTeZmHrfaNBAAdQKEvxl8NaU6Ce4YoaOGVQ== 2024-09-15T02:58:11.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDTeZmHrfaNBAAdQKEvxl8NaU6Ce4YoaOGVQ=="}' 2024-09-15T02:58:11.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2df72312-329d-42b8-a241-324d5238dbb0 -i td/test-erasure-eio/3/new.json 2024-09-15T02:58:13.209 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:58:13.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T02:58:13.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDTeZmHrfaNBAAdQKEvxl8NaU6Ce4YoaOGVQ== --osd-uuid 2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T02:58:13.245 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:13.245+0000 7f98f420e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:13.259 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:13.258+0000 7f98f420e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:13.260 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:13.259+0000 7f98f420e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:13.260 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:13.259+0000 7f98f420e440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:58:13.260 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:13.259+0000 7f98f420e440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:58:13.260 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:13.259+0000 7f98f420e440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T02:58:13.367 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:13.366+0000 7f98f420e440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T02:58:19.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T02:58:19.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T02:58:19.322 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T02:58:19.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T02:58:19.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T02:58:19.826 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T02:58:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T02:58:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:58:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T02:58:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T02:58:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T02:58:19.849 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:19.848+0000 7f740900a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:19.885 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:19.884+0000 7f740900a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:19.886 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:19.885+0000 7f740900a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:58:20.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T02:58:20.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:58:20.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:58:20.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:58:20.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:58:20.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:20.206 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T02:58:20.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:58:20.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:20.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:20.404 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:20.403+0000 7f740900a440 -1 Falling back to public interface 2024-09-15T02:58:20.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:21.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:21.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:21.573 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T02:58:21.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:58:21.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:21.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:21.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:22.871 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:58:22.870+0000 7f740900a440 -1 osd.3 0 log_to_monitors true 2024-09-15T02:58:22.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:22.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:22.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:58:22.933 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T02:58:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:23.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:24.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:24.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:24.576 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T02:58:24.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:58:24.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:24.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:25.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:26.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:26.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:26.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T02:58:26.158 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T02:58:26.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:26.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:26.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:27.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:27.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:27.641 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T02:58:27.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T02:58:27.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:27.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:28.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:29.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:29.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:29.071 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T02:58:29.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T02:58:29.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:29.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:29.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:30.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:30.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:30.469 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T02:58:30.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T02:58:30.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:30.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:30.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:31.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:31.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:31.970 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T02:58:31.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T02:58:31.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:31.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:32.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:33.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:33.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:33.365 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T02:58:33.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T02:58:33.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:33.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:33.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:34.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:34.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:34.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T02:58:34.771 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T02:58:34.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:34.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:35.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:36.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:36.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:36.254 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T02:58:36.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T02:58:36.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:36.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:37.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:38.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:38.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:38.254 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T02:58:38.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T02:58:38.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:38.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:39.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:40.139 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T02:58:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T02:58:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:40.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:40.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:41.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:41.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:41.870 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T02:58:41.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T02:58:41.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:41.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:42.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:43.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:43.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:43.682 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T02:58:43.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T02:58:43.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:43.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:44.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:45.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:45.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:45.526 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T02:58:45.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T02:58:45.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:45.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:46.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:58:47.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:58:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:58:47.411 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T02:58:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T02:58:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:58:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:58:47.906 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/208087,v1:127.0.0.1:6827/208087] [v2:127.0.0.1:6828/208087,v1:127.0.0.1:6829/208087] exists,up 2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T02:58:47.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:58:47.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:58:47.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:58:47.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T02:58:47.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T02:58:47.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T02:58:47.908 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T02:58:47.908 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:58:47.908 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:58:47.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T02:58:47.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T02:58:47.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T02:58:47.978 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T02:58:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T02:58:47.988 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T02:56:36.257+0000 7f147520f440 0 load: jerasure load: lrc 2024-09-15T02:58:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-09-15T02:58:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T02:58:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T02:58:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T02:58:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T02:58:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T02:58:47.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T02:58:47.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T02:58:47.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T02:58:48.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T02:58:48.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T02:58:49.996 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T02:58:50.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T02:58:51.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T02:58:51.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:58:51.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:58:51.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:58:51.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:58:51.011 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:58:51.011 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:58:51.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:58:51.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:58:51.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:58:51.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:58:51.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:58:51.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:58:51.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:58:51.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:58:51.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:58:51.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:58:51.481 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:58:51.481 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:58:51.481 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T02:58:51.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:58:51.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:58:51.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:58:51.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803806 2024-09-15T02:58:51.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803806 2024-09-15T02:58:51.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806' 2024-09-15T02:58:51.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:58:51.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:58:51.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574867 2024-09-15T02:58:51.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574867 2024-09-15T02:58:51.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-55834574867' 2024-09-15T02:58:51.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:58:51.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:58:51.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378639 2024-09-15T02:58:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378639 2024-09-15T02:58:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-55834574867 2-81604378639' 2024-09-15T02:58:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:58:51.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:58:51.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051591 2024-09-15T02:58:51.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051591 2024-09-15T02:58:51.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-55834574867 2-81604378639 3-124554051591' 2024-09-15T02:58:51.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:58:51.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803806 2024-09-15T02:58:51.917 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:58:51.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:58:51.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803806 2024-09-15T02:58:51.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:58:51.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803806 2024-09-15T02:58:51.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-09-15T02:58:51.919 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803806 2024-09-15T02:58:51.920 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:58:52.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803804 -lt 25769803806 2024-09-15T02:58:52.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:58:53.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:58:53.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:58:53.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803807 -lt 25769803806 2024-09-15T02:58:53.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:58:53.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574867 2024-09-15T02:58:53.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:58:53.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:58:53.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574867 2024-09-15T02:58:53.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:58:53.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574867 2024-09-15T02:58:53.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574867' 2024-09-15T02:58:53.631 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574867 2024-09-15T02:58:53.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:58:54.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574868 -lt 55834574867 2024-09-15T02:58:54.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:58:54.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378639 2024-09-15T02:58:54.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:58:54.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:58:54.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378639 2024-09-15T02:58:54.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:58:54.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378639 2024-09-15T02:58:54.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2024-09-15T02:58:54.014 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378639 2024-09-15T02:58:54.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:58:54.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378640 -lt 81604378639 2024-09-15T02:58:54.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:58:54.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051591 2024-09-15T02:58:54.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:58:54.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:58:54.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051591 2024-09-15T02:58:54.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:58:54.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051591 2024-09-15T02:58:54.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051591' 2024-09-15T02:58:54.364 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 124554051591 2024-09-15T02:58:54.364 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:58:54.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051592 -lt 124554051591 2024-09-15T02:58:54.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:58:54.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:58:54.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:58:55.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:58:55.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:58:55.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:58:55.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:58:55.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:58:56.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:58:56.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:58:56.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:58:56.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-09-15T02:58:56.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-09-15T02:58:56.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T02:58:56.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-15T02:58:56.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-15T02:58:56.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-15T02:58:56.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-09-15T02:58:56.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-15T02:58:56.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-15T02:58:56.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T02:58:56.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-87967-0-10 2024-09-15T02:58:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87967-0-10 2024-09-15T02:58:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:58:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:58:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87967-0-10 2024-09-15T02:58:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:58:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:58:56.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:58:56.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:58:56.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:58:56.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:58:56.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:58:56.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:58:56.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87967-0-10 td/test-erasure-eio/ORIGINAL 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-87967-0-10 td/test-erasure-eio 0 10 set 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-0-10 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T02:58:56.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T02:58:56.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-0-10 2024-09-15T02:58:56.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:58:56.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-0-10 2024-09-15T02:58:56.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-0-10 2024-09-15T02:58:56.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:58:56.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:58:56.593 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:58:56.593 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:58:56.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T02:58:56.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T02:58:56.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T02:58:56.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-15T02:58:56.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T02:58:57.242 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T02:58:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T02:58:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-15T02:58:57.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T02:58:57.254 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:10 bytes copied, 5.6768e-05 s, 176 kB/s 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-15T02:58:57.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:58:57.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T02:58:57.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:58:57.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:58:57.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:58:57.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:58:57.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:58:58.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:58:58.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:58:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T02:58:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:58:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-15T02:58:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:58:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T02:58:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:00.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:59:00.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:59:00.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T02:59:00.515 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.3 2024-09-15T02:59:00.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:59:00.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T02:59:00.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T02:59:00.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:59:00.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:59:00.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:59:00.538 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:00.537+0000 7fc91c11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:00.540 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:00.540+0000 7fc91c11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:00.542 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:00.541+0000 7fc91c11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:00.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:01.062 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:01.061+0000 7fc91c11d440 -1 Falling back to public interface 2024-09-15T02:59:01.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:02.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:03.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:03.855 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:03.854+0000 7fc91c11d440 -1 osd.3 39 log_to_monitors true 2024-09-15T02:59:03.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:59:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:05.502 INFO:tasks.workunit.client.0.smithi130.stderr:osd.3 up in weight 1 up_from 44 up_thru 44 down_at 41 last_clean_interval [29,39) [v2:127.0.0.1:6826/209917,v1:127.0.0.1:6827/209917] [v2:127.0.0.1:6828/209917,v1:127.0.0.1:6829/209917] exists,up 2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T02:59:05.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:05.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:05.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:05.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:59:05.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:59:05.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:59:05.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:59:05.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:59:05.504 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:59:05.504 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:59:05.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:59:05.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:59:05.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:59:05.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:59:05.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:59:05.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:59:05.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:59:05.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:59:05.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:59:05.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:59:05.942 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:05.942 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:05.942 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T02:59:05.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:59:05.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:05.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:59:06.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803811 2024-09-15T02:59:06.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803811 2024-09-15T02:59:06.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811' 2024-09-15T02:59:06.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:06.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:59:06.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574872 2024-09-15T02:59:06.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574872 2024-09-15T02:59:06.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574872' 2024-09-15T02:59:06.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:06.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:59:06.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378644 2024-09-15T02:59:06.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378644 2024-09-15T02:59:06.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574872 2-81604378644' 2024-09-15T02:59:06.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:06.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:59:06.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561027 2024-09-15T02:59:06.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561027 2024-09-15T02:59:06.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574872 2-81604378644 3-188978561027' 2024-09-15T02:59:06.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:06.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803811 2024-09-15T02:59:06.397 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:06.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:59:06.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803811 2024-09-15T02:59:06.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:06.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803811 2024-09-15T02:59:06.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-15T02:59:06.400 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803811 2024-09-15T02:59:06.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:59:06.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-15T02:59:06.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:59:07.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:59:07.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:59:08.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803811 -lt 25769803811 2024-09-15T02:59:08.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:08.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574872 2024-09-15T02:59:08.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:08.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:59:08.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574872 2024-09-15T02:59:08.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:08.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574872 2024-09-15T02:59:08.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574872' 2024-09-15T02:59:08.107 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 55834574872 2024-09-15T02:59:08.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:59:08.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574873 -lt 55834574872 2024-09-15T02:59:08.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:08.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378644 2024-09-15T02:59:08.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:08.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:59:08.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378644 2024-09-15T02:59:08.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:08.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378644 2024-09-15T02:59:08.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-09-15T02:59:08.469 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378644 2024-09-15T02:59:08.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:59:08.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378644 -lt 81604378644 2024-09-15T02:59:08.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:08.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-188978561027 2024-09-15T02:59:08.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:08.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:59:08.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-188978561027 2024-09-15T02:59:08.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:08.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561027 2024-09-15T02:59:08.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 188978561027' 2024-09-15T02:59:08.818 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 188978561027 2024-09-15T02:59:08.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:59:09.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561027 -lt 188978561027 2024-09-15T02:59:09.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:59:09.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:09.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:09.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:59:09.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:59:09.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:59:09.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:59:09.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:59:09.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:59:09.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:59:09.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:59:09.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:59:09.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:59:09.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:09.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:10.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:59:10.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:59:10.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:59:10.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T02:59:10.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T02:59:11.219 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T02:59:11.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-0-10 2024-09-15T02:59:11.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:59:11.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:59:11.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-0-10 2024-09-15T02:59:11.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:59:11.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:59:11.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87967-0-10 td/test-erasure-eio/COPY 2024-09-15T02:59:11.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:59:11.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:59:11.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-15T02:59:11.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-87967-0-10 td/test-erasure-eio 1 10 set 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-0-10 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T02:59:11.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T02:59:11.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T02:59:11.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-0-10 2024-09-15T02:59:11.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:59:11.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-0-10 2024-09-15T02:59:11.342 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-0-10 2024-09-15T02:59:11.342 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:59:11.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:59:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:59:11.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T02:59:11.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T02:59:11.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T02:59:11.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-15T02:59:11.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T02:59:12.315 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T02:59:12.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T02:59:12.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-15T02:59:12.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T02:59:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T02:59:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T02:59:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:10 bytes copied, 4.5541e-05 s, 220 kB/s 2024-09-15T02:59:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:12.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T02:59:12.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:59:12.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T02:59:12.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:59:12.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:12.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T02:59:12.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:59:12.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T02:59:12.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:59:12.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T02:59:12.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:12.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:12.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:12.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:12.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:12.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:12.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:12.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T02:59:12.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:59:12.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T02:59:12.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:59:12.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T02:59:12.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:15.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T02:59:15.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:59:15.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:59:15.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:59:15.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:15.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:15.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:15.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:59:15.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:59:15.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:59:15.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T02:59:15.097 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T02:59:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:59:15.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T02:59:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T02:59:15.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:59:15.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:59:15.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:59:15.119 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:15.118+0000 7f4db651c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:15.135 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:15.135+0000 7f4db651c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:15.137 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:15.136+0000 7f4db651c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:59:15.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:15.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:15.656 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:15.655+0000 7f4db651c440 -1 Falling back to public interface 2024-09-15T02:59:15.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:16.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:16.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:16.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:16.851 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:16.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:16.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:17.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:18.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:18.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:18.226 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:18.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:18.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:18.238 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:18.237+0000 7f4db651c440 -1 osd.1 48 log_to_monitors true 2024-09-15T02:59:18.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:19.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:19.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:19.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:19.675 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:59:19.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:19.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:20.248 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [13,48) [v2:127.0.0.1:6810/211369,v1:127.0.0.1:6811/211369] [v2:127.0.0.1:6812/211369,v1:127.0.0.1:6813/211369] exists,up 14174469-0503-4543-a4e2-260af417188e 2024-09-15T02:59:20.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:20.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:20.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:20.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:59:20.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:59:20.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:59:20.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:59:20.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:59:20.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:59:20.250 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:59:20.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:59:20.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:59:20.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:59:20.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:59:20.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:59:20.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:59:20.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:59:20.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:59:20.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:59:20.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:59:20.690 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:20.690 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:20.690 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T02:59:20.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:59:20.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:20.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:59:20.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803816 2024-09-15T02:59:20.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803816 2024-09-15T02:59:20.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816' 2024-09-15T02:59:20.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:20.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:59:20.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201283 2024-09-15T02:59:20.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201283 2024-09-15T02:59:20.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-236223201283' 2024-09-15T02:59:20.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:20.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:59:21.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378649 2024-09-15T02:59:21.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378649 2024-09-15T02:59:21.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-236223201283 2-81604378649' 2024-09-15T02:59:21.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:21.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:59:21.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561032 2024-09-15T02:59:21.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561032 2024-09-15T02:59:21.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-236223201283 2-81604378649 3-188978561032' 2024-09-15T02:59:21.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:21.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803816 2024-09-15T02:59:21.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:21.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:59:21.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803816 2024-09-15T02:59:21.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:21.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803816 2024-09-15T02:59:21.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-15T02:59:21.129 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803816 2024-09-15T02:59:21.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:59:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803816 -lt 25769803816 2024-09-15T02:59:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:21.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-236223201283 2024-09-15T02:59:21.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:21.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:59:21.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-236223201283 2024-09-15T02:59:21.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201283 2024-09-15T02:59:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 236223201283' 2024-09-15T02:59:21.547 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 236223201283 2024-09-15T02:59:21.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:59:21.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201283 -lt 236223201283 2024-09-15T02:59:21.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:21.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378649 2024-09-15T02:59:21.897 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:21.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:59:21.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378649 2024-09-15T02:59:21.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:21.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378649 2024-09-15T02:59:21.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378649' 2024-09-15T02:59:21.899 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378649 2024-09-15T02:59:21.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:59:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378647 -lt 81604378649 2024-09-15T02:59:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:59:23.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:59:23.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:59:23.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378650 -lt 81604378649 2024-09-15T02:59:23.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:23.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-188978561032 2024-09-15T02:59:23.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:23.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:59:23.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-188978561032 2024-09-15T02:59:23.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:23.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561032 2024-09-15T02:59:23.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 188978561032' 2024-09-15T02:59:23.638 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 188978561032 2024-09-15T02:59:23.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:59:23.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561032 -lt 188978561032 2024-09-15T02:59:23.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:59:23.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:23.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:24.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:59:24.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:59:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:59:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:59:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:59:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:59:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:59:24.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:59:24.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:59:24.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:59:24.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:24.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:25.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:59:25.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:59:25.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:59:25.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T02:59:25.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T02:59:26.050 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T02:59:26.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-0-10 fail 2024-09-15T02:59:26.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:59:26.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:59:26.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-0-10 2024-09-15T02:59:26.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T02:59:26.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T02:59:26.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87967-0-10 td/test-erasure-eio/COPY 2024-09-15T02:59:26.220 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-size-87967-0-10: (5) Input/output error 2024-09-15T02:59:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T02:59:26.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T02:59:26.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-87967-0-0 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87967-0-0 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:59:26.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87967-0-0 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:59:26.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87967-0-0 td/test-erasure-eio/ORIGINAL 2024-09-15T02:59:26.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-87967-0-0 td/test-erasure-eio 0 0 set 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-0-0 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-0-0 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-0-0 2024-09-15T02:59:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-0-0 2024-09-15T02:59:26.389 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:59:26.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:59:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:26.779 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:59:26.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T02:59:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T02:59:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T02:59:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-15T02:59:26.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T02:59:27.414 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T02:59:27.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T02:59:27.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-15T02:59:27.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T02:59:27.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:27.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T02:59:27.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:59:27.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-15T02:59:27.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:27.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:27.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:27.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:27.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:27.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T02:59:27.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:59:27.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-15T02:59:27.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:59:27.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T02:59:27.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:30.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T02:59:30.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:59:30.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:30.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:30.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.3 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:59:30.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T02:59:30.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T02:59:30.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:59:30.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:59:30.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:59:30.177 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:30.176+0000 7f5d0cb28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:30.179 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:30.178+0000 7f5d0cb28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:30.181 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:30.180+0000 7f5d0cb28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:30.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:30.697 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:30.696+0000 7f5d0cb28440 -1 Falling back to public interface 2024-09-15T02:59:30.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:31.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:31.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:31.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:31.860 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:31.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:31.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:32.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:33.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:33.242 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:33.241+0000 7f5d0cb28440 -1 osd.3 59 log_to_monitors true 2024-09-15T02:59:33.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:34.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:34.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:34.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:34.598 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:59:34.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:34.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:osd.3 up in weight 1 up_from 65 up_thru 65 down_at 62 last_clean_interval [44,59) [v2:127.0.0.1:6826/212839,v1:127.0.0.1:6827/212839] [v2:127.0.0.1:6828/212839,v1:127.0.0.1:6829/212839] exists,up 2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T02:59:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:59:34.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:59:34.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:59:35.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:59:35.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:59:35.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:59:35.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:59:35.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:59:35.048 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:59:35.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:59:35.403 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:35.403 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:35.403 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T02:59:35.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:59:35.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:35.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:59:35.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803821 2024-09-15T02:59:35.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803821 2024-09-15T02:59:35.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821' 2024-09-15T02:59:35.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:35.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:59:35.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201288 2024-09-15T02:59:35.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201288 2024-09-15T02:59:35.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-236223201288' 2024-09-15T02:59:35.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:35.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:59:35.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378654 2024-09-15T02:59:35.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378654 2024-09-15T02:59:35.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-236223201288 2-81604378654' 2024-09-15T02:59:35.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:35.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:59:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=279172874243 2024-09-15T02:59:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 279172874243 2024-09-15T02:59:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-236223201288 2-81604378654 3-279172874243' 2024-09-15T02:59:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:35.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803821 2024-09-15T02:59:35.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:59:35.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803821 2024-09-15T02:59:35.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:35.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803821 2024-09-15T02:59:35.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803821' 2024-09-15T02:59:35.897 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803821 2024-09-15T02:59:35.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:59:36.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803819 -lt 25769803821 2024-09-15T02:59:36.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:59:37.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:59:37.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:59:37.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803821 -lt 25769803821 2024-09-15T02:59:37.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:37.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-236223201288 2024-09-15T02:59:37.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:37.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:59:37.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-236223201288 2024-09-15T02:59:37.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:37.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201288 2024-09-15T02:59:37.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 236223201288' 2024-09-15T02:59:37.603 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 236223201288 2024-09-15T02:59:37.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:59:37.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201288 -lt 236223201288 2024-09-15T02:59:37.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:37.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378654 2024-09-15T02:59:37.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:37.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:59:37.994 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378654 2024-09-15T02:59:37.994 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:37.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378654 2024-09-15T02:59:37.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378654' 2024-09-15T02:59:37.995 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378654 2024-09-15T02:59:37.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:59:38.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378654 -lt 81604378654 2024-09-15T02:59:38.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:38.344 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-279172874243 2024-09-15T02:59:38.344 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:38.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:59:38.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-279172874243 2024-09-15T02:59:38.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:38.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=279172874243 2024-09-15T02:59:38.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 279172874243' 2024-09-15T02:59:38.346 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 279172874243 2024-09-15T02:59:38.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:59:38.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874243 -lt 279172874243 2024-09-15T02:59:38.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:59:38.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:38.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:59:39.196 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:59:39.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:59:39.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:59:39.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:39.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:59:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:59:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:59:40.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T02:59:40.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T02:59:40.604 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T02:59:40.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-0-0 2024-09-15T02:59:40.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:59:40.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:59:40.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-0-0 2024-09-15T02:59:40.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T02:59:40.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T02:59:40.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87967-0-0 td/test-erasure-eio/COPY 2024-09-15T02:59:40.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T02:59:40.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T02:59:40.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-15T02:59:40.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-15T02:59:40.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-87967-0-0 td/test-erasure-eio 1 0 set 2024-09-15T02:59:40.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-0-0 2024-09-15T02:59:40.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T02:59:40.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T02:59:40.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-0-0 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-0-0 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-0-0 2024-09-15T02:59:40.741 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:59:41.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:59:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:59:41.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T02:59:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T02:59:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T02:59:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-15T02:59:41.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T02:59:41.755 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T02:59:41.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T02:59:41.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-15T02:59:41.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T02:59:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T02:59:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:59:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T02:59:41.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:41.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:41.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:41.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T02:59:42.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:44.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:44.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:59:44.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T02:59:44.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T02:59:44.637 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T02:59:44.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:59:44.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T02:59:44.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T02:59:44.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:59:44.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:59:44.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:59:44.660 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:44.658+0000 7f303920f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:44.662 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:44.660+0000 7f303920f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:44.664 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:44.662+0000 7f303920f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:44.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T02:59:44.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:44.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T02:59:44.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:44.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:44.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:44.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:44.996 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:59:44.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:44.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:45.178 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:45.176+0000 7f303920f440 -1 Falling back to public interface 2024-09-15T02:59:45.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:46.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:46.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:46.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T02:59:46.355 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:46.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:46.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:46.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T02:59:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:47.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:47.759 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:47.757+0000 7f303920f440 -1 osd.1 71 log_to_monitors true 2024-09-15T02:59:48.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T02:59:49.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T02:59:49.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:49.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T02:59:49.137 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T02:59:49.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:49.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T02:59:49.584 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 76 up_thru 76 down_at 72 last_clean_interval [55,71) [v2:127.0.0.1:6810/214291,v1:127.0.0.1:6811/214291] [v2:127.0.0.1:6812/214291,v1:127.0.0.1:6813/214291] exists,up 14174469-0503-4543-a4e2-260af417188e 2024-09-15T02:59:49.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T02:59:49.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T02:59:49.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T02:59:49.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T02:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T02:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T02:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T02:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T02:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T02:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T02:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T02:59:49.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T02:59:49.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T02:59:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T02:59:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T02:59:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T02:59:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T02:59:49.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T02:59:49.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T02:59:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T02:59:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T02:59:50.037 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T02:59:50.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T02:59:50.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:50.038 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T02:59:50.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803826 2024-09-15T02:59:50.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803826 2024-09-15T02:59:50.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826' 2024-09-15T02:59:50.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:50.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T02:59:50.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=326417514499 2024-09-15T02:59:50.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 326417514499 2024-09-15T02:59:50.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-326417514499' 2024-09-15T02:59:50.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:50.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T02:59:50.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378659 2024-09-15T02:59:50.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378659 2024-09-15T02:59:50.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-326417514499 2-81604378659' 2024-09-15T02:59:50.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T02:59:50.374 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T02:59:50.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=279172874248 2024-09-15T02:59:50.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 279172874248 2024-09-15T02:59:50.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-326417514499 2-81604378659 3-279172874248' 2024-09-15T02:59:50.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:50.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803826 2024-09-15T02:59:50.479 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:50.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T02:59:50.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803826 2024-09-15T02:59:50.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:50.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803826 2024-09-15T02:59:50.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-09-15T02:59:50.481 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803826 2024-09-15T02:59:50.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:59:50.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803824 -lt 25769803826 2024-09-15T02:59:50.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T02:59:51.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T02:59:51.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T02:59:52.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803826 -lt 25769803826 2024-09-15T02:59:52.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:52.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-326417514499 2024-09-15T02:59:52.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:52.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T02:59:52.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-326417514499 2024-09-15T02:59:52.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:52.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=326417514499 2024-09-15T02:59:52.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2024-09-15T02:59:52.185 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 326417514499 2024-09-15T02:59:52.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T02:59:52.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 326417514499 -lt 326417514499 2024-09-15T02:59:52.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:52.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378659 2024-09-15T02:59:52.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:52.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T02:59:52.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378659 2024-09-15T02:59:52.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:52.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378659 2024-09-15T02:59:52.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378659' 2024-09-15T02:59:52.553 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378659 2024-09-15T02:59:52.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T02:59:52.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378659 -lt 81604378659 2024-09-15T02:59:52.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T02:59:52.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-279172874248 2024-09-15T02:59:52.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T02:59:52.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T02:59:52.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-279172874248 2024-09-15T02:59:52.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T02:59:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=279172874248 2024-09-15T02:59:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 279172874248' 2024-09-15T02:59:52.904 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 279172874248 2024-09-15T02:59:52.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T02:59:53.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 279172874248 -lt 279172874248 2024-09-15T02:59:53.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T02:59:53.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:53.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:53.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T02:59:53.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T02:59:53.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T02:59:53.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T02:59:53.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T02:59:53.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T02:59:53.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T02:59:53.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T02:59:54.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T02:59:54.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T02:59:54.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T02:59:54.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T02:59:54.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T02:59:54.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T02:59:54.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T02:59:54.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T02:59:54.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T02:59:55.220 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T02:59:55.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-0-0 fail 2024-09-15T02:59:55.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T02:59:55.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T02:59:55.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-0-0 2024-09-15T02:59:55.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T02:59:55.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T02:59:55.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87967-0-0 td/test-erasure-eio/COPY 2024-09-15T02:59:55.362 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-size-87967-0-0: (5) Input/output error 2024-09-15T02:59:55.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T02:59:55.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T02:59:55.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-87967-0-256 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87967-0-256 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87967-0-256 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T02:59:55.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T02:59:55.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87967-0-256 td/test-erasure-eio/ORIGINAL 2024-09-15T02:59:55.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-87967-0-256 td/test-erasure-eio 0 256 add 2024-09-15T02:59:55.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-0-256 2024-09-15T02:59:55.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T02:59:55.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-0-256 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-0-256 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-0-256 2024-09-15T02:59:55.534 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T02:59:55.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T02:59:55.953 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T02:59:55.953 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T02:59:55.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T02:59:55.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T02:59:55.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T02:59:55.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-15T02:59:55.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T02:59:56.584 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T02:59:56.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-15T02:59:56.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T02:59:56.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T02:59:56.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T02:59:56.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T02:59:56.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T02:59:59.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T02:59:59.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T02:59:59.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T02:59:59.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T02:59:59.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T02:59:59.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T02:59:59.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T02:59:59.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T02:59:59.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T02:59:59.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T02:59:59.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T02:59:59.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T02:59:59.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:59.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T02:59:59.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T02:59:59.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.3 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T02:59:59.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T02:59:59.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T02:59:59.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T02:59:59.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T02:59:59.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T02:59:59.252 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:59.251+0000 7fe23b00a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:59.254 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:59.253+0000 7fe23b00a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:59.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:59.254+0000 7fe23b00a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T02:59:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T02:59:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T02:59:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T02:59:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T02:59:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T02:59:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T02:59:59.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T02:59:59.604 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T02:59:59.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T02:59:59.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T02:59:59.768 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T02:59:59.767+0000 7fe23b00a440 -1 Falling back to public interface 2024-09-15T02:59:59.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:00.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:00.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:00.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:00:00.954 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:00.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:00.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:00:01.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:02.265 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:02.264+0000 7fe23b00a440 -1 osd.3 81 log_to_monitors true 2024-09-15T03:00:02.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:02.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:02.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:00:02.322 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:02.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:02.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:00:02.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:03.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:03.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:03.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:00:03.723 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:00:03.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:03.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:00:04.221 INFO:tasks.workunit.client.0.smithi130.stderr:osd.3 up in weight 1 up_from 86 up_thru 86 down_at 83 last_clean_interval [65,81) [v2:127.0.0.1:6826/215760,v1:127.0.0.1:6827/215760] [v2:127.0.0.1:6828/215760,v1:127.0.0.1:6829/215760] exists,up 2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T03:00:04.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:00:04.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:00:04.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:00:04.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:00:04.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:00:04.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:00:04.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:00:04.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:00:04.223 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:00:04.223 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:00:04.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:00:04.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:00:04.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:00:04.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:00:04.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:00:04.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:00:04.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:00:04.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:00:04.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:00:04.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:00:04.657 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:04.657 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:04.657 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:00:04.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:00:04.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:04.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:00:04.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803831 2024-09-15T03:00:04.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803831 2024-09-15T03:00:04.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831' 2024-09-15T03:00:04.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:04.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:00:04.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=326417514504 2024-09-15T03:00:04.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 326417514504 2024-09-15T03:00:04.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-326417514504' 2024-09-15T03:00:04.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:04.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:00:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378664 2024-09-15T03:00:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378664 2024-09-15T03:00:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-326417514504 2-81604378664' 2024-09-15T03:00:04.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:04.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:00:05.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=369367187459 2024-09-15T03:00:05.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 369367187459 2024-09-15T03:00:05.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-326417514504 2-81604378664 3-369367187459' 2024-09-15T03:00:05.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:05.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803831 2024-09-15T03:00:05.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:05.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:00:05.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803831 2024-09-15T03:00:05.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:05.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803831 2024-09-15T03:00:05.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-15T03:00:05.106 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803831 2024-09-15T03:00:05.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:05.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-15T03:00:05.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:05.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-326417514504 2024-09-15T03:00:05.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:05.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:00:05.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-326417514504 2024-09-15T03:00:05.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:05.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=326417514504 2024-09-15T03:00:05.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 326417514504' 2024-09-15T03:00:05.454 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 326417514504 2024-09-15T03:00:05.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:00:05.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 326417514504 -lt 326417514504 2024-09-15T03:00:05.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:05.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378664 2024-09-15T03:00:05.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:05.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:00:05.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378664 2024-09-15T03:00:05.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:05.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378664 2024-09-15T03:00:05.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378664' 2024-09-15T03:00:05.807 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378664 2024-09-15T03:00:05.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:00:06.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378664 -lt 81604378664 2024-09-15T03:00:06.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:06.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-369367187459 2024-09-15T03:00:06.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:06.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:00:06.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-369367187459 2024-09-15T03:00:06.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:06.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=369367187459 2024-09-15T03:00:06.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 369367187459' 2024-09-15T03:00:06.157 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 369367187459 2024-09-15T03:00:06.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:00:06.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 369367187459 2024-09-15T03:00:06.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:00:07.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:00:07.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:00:07.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 369367187459 -lt 369367187459 2024-09-15T03:00:07.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:00:07.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:07.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:00:08.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:00:08.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:00:08.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:00:08.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:08.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:09.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:00:09.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:00:09.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:00:09.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T03:00:09.159 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T03:00:09.159 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T03:00:09.159 INFO:tasks.workunit.client.0.smithi130.stderr:256 bytes copied, 3.3684e-05 s, 7.6 MB/s 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-15T03:00:09.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:09.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:09.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:09.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:09.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-15T03:00:09.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:00:11.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:00:11.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T03:00:11.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T03:00:11.748 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.3 2024-09-15T03:00:11.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:00:11.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-15T03:00:11.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T03:00:11.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:00:11.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:00:11.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:00:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:11.769+0000 7f4ccd489440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:11.773 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:11.772+0000 7f4ccd489440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:11.774 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:11.773+0000 7f4ccd489440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:12.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:00:12.292 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:12.291+0000 7f4ccd489440 -1 Falling back to public interface 2024-09-15T03:00:12.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:00:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:13.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:00:13.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:14.767 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:14.766+0000 7f4ccd489440 -1 osd.3 88 log_to_monitors true 2024-09-15T03:00:14.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:14.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:14.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:00:14.818 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:14.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:14.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:00:15.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:16.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:16.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:16.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:00:16.249 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:00:16.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:16.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:00:16.657 INFO:tasks.workunit.client.0.smithi130.stderr:osd.3 up in weight 1 up_from 92 up_thru 92 down_at 89 last_clean_interval [86,88) [v2:127.0.0.1:6826/217118,v1:127.0.0.1:6827/217118] [v2:127.0.0.1:6828/217118,v1:127.0.0.1:6829/217118] exists,up 2df72312-329d-42b8-a241-324d5238dbb0 2024-09-15T03:00:16.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:00:16.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:00:16.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:00:16.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:00:16.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:00:16.658 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:00:16.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:00:16.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:00:16.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:00:16.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:00:16.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:00:16.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:00:17.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:00:17.140 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:17.141 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:17.141 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:00:17.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:00:17.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:17.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:00:17.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803836 2024-09-15T03:00:17.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803836 2024-09-15T03:00:17.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836' 2024-09-15T03:00:17.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:17.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:00:17.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=326417514508 2024-09-15T03:00:17.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 326417514508 2024-09-15T03:00:17.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-326417514508' 2024-09-15T03:00:17.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:17.383 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:00:17.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378669 2024-09-15T03:00:17.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378669 2024-09-15T03:00:17.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-326417514508 2-81604378669' 2024-09-15T03:00:17.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:17.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:00:17.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=395136991235 2024-09-15T03:00:17.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 395136991235 2024-09-15T03:00:17.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-326417514508 2-81604378669 3-395136991235' 2024-09-15T03:00:17.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:17.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803836 2024-09-15T03:00:17.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:17.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:00:17.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803836 2024-09-15T03:00:17.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:17.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803836 2024-09-15T03:00:17.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-09-15T03:00:17.627 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803836 2024-09-15T03:00:17.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:17.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803833 -lt 25769803836 2024-09-15T03:00:17.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:00:18.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:00:18.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:19.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803836 -lt 25769803836 2024-09-15T03:00:19.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:19.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-326417514508 2024-09-15T03:00:19.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:19.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:00:19.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-326417514508 2024-09-15T03:00:19.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:19.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=326417514508 2024-09-15T03:00:19.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 326417514508' 2024-09-15T03:00:19.390 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 326417514508 2024-09-15T03:00:19.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:00:19.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 326417514509 -lt 326417514508 2024-09-15T03:00:19.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:19.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378669 2024-09-15T03:00:19.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:19.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:00:19.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378669 2024-09-15T03:00:19.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:19.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378669 2024-09-15T03:00:19.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378669' 2024-09-15T03:00:19.740 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378669 2024-09-15T03:00:19.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:00:20.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378669 -lt 81604378669 2024-09-15T03:00:20.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:20.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-395136991235 2024-09-15T03:00:20.102 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:20.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:00:20.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-395136991235 2024-09-15T03:00:20.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:20.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=395136991235 2024-09-15T03:00:20.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-09-15T03:00:20.104 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 395136991235 2024-09-15T03:00:20.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:00:20.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 395136991235 -lt 395136991235 2024-09-15T03:00:20.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:00:20.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:20.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:20.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:00:20.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:00:20.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:00:20.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:00:20.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:00:20.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:00:20.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:00:20.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:00:21.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:00:21.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:00:21.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:21.268 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:00:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:00:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:00:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:00:21.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:00:22.410 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:00:22.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-0-256 2024-09-15T03:00:22.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:00:22.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:00:22.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-0-256 2024-09-15T03:00:22.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:00:22.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:00:22.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87967-0-256 td/test-erasure-eio/COPY 2024-09-15T03:00:22.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:00:22.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:00:22.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-15T03:00:22.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-15T03:00:22.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-87967-0-256 td/test-erasure-eio 1 256 add 2024-09-15T03:00:22.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-0-256 2024-09-15T03:00:22.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T03:00:22.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T03:00:22.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T03:00:22.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-15T03:00:22.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T03:00:22.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-15T03:00:22.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T03:00:22.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-15T03:00:22.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T03:00:22.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-0-256 2024-09-15T03:00:22.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:00:22.632 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-0-256 2024-09-15T03:00:22.632 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-0-256 2024-09-15T03:00:22.633 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-15T03:00:23.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T03:00:23.673 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T03:00:23.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:00:23.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T03:00:23.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:23.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:23.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:23.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:23.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:00:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:00:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:00:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:00:23.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T03:00:23.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:26.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T03:00:26.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:00:26.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:00:26.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:00:26.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:00:26.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:26.671 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:26.672 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:26.672 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:26.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:26.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:26.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:00:26.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:00:26.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:00:26.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T03:00:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:00:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:00:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:00:26.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:00:26.694 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:26.693+0000 7efe21f27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:26.696 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:26.695+0000 7efe21f27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:26.698 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:26.697+0000 7efe21f27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:27.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:00:27.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:00:27.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:00:27.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:00:27.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:00:27.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:27.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:00:27.030 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:00:27.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:27.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:27.217 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:27.216+0000 7efe21f27440 -1 Falling back to public interface 2024-09-15T03:00:27.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:28.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:28.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:28.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:00:28.388 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:28.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:28.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:28.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:29.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:29.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:29.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:00:29.745 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:29.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:29.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:29.811 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:29.810+0000 7efe21f27440 -1 osd.1 96 log_to_monitors true 2024-09-15T03:00:30.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:31.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:31.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:31.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:00:31.292 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:00:31.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:31.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 103 up_thru 103 down_at 99 last_clean_interval [76,96) [v2:127.0.0.1:6810/218570,v1:127.0.0.1:6811/218570] [v2:127.0.0.1:6812/218570,v1:127.0.0.1:6813/218570] exists,up 14174469-0503-4543-a4e2-260af417188e 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:00:31.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:00:31.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:00:31.748 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:00:31.748 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:00:31.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:00:31.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:00:31.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:00:31.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:00:31.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:00:31.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:00:31.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:00:31.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:00:31.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:00:32.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:00:32.186 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:32.186 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:32.186 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:00:32.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:00:32.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:32.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:00:32.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803841 2024-09-15T03:00:32.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803841 2024-09-15T03:00:32.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803841' 2024-09-15T03:00:32.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:32.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:00:32.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=442381631491 2024-09-15T03:00:32.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 442381631491 2024-09-15T03:00:32.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803841 1-442381631491' 2024-09-15T03:00:32.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:32.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:00:32.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378674 2024-09-15T03:00:32.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378674 2024-09-15T03:00:32.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803841 1-442381631491 2-81604378674' 2024-09-15T03:00:32.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:32.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:00:32.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=395136991240 2024-09-15T03:00:32.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 395136991240 2024-09-15T03:00:32.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803841 1-442381631491 2-81604378674 3-395136991240' 2024-09-15T03:00:32.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:32.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803841 2024-09-15T03:00:32.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:32.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:00:32.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803841 2024-09-15T03:00:32.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:32.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803841 2024-09-15T03:00:32.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-09-15T03:00:32.618 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803841 2024-09-15T03:00:32.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:32.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803838 -lt 25769803841 2024-09-15T03:00:32.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:00:33.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:00:33.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:34.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803841 -lt 25769803841 2024-09-15T03:00:34.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:34.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-442381631491 2024-09-15T03:00:34.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:34.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:00:34.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-442381631491 2024-09-15T03:00:34.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:34.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=442381631491 2024-09-15T03:00:34.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 442381631491' 2024-09-15T03:00:34.325 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 442381631491 2024-09-15T03:00:34.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 442381631491 -lt 442381631491 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378674 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378674 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378674 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378674' 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378674 2024-09-15T03:00:34.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:00:35.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378674 -lt 81604378674 2024-09-15T03:00:35.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:35.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-395136991240 2024-09-15T03:00:35.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:35.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:00:35.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-395136991240 2024-09-15T03:00:35.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:35.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=395136991240 2024-09-15T03:00:35.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 395136991240' 2024-09-15T03:00:35.044 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 395136991240 2024-09-15T03:00:35.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:00:35.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 395136991240 -lt 395136991240 2024-09-15T03:00:35.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:00:35.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:35.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:35.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:00:35.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:00:35.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:00:35.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:00:35.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:00:35.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:00:35.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:00:35.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:00:36.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:00:36.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:00:36.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:36.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:36.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:00:36.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:00:36.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:00:36.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:256 bytes copied, 3.2094e-05 s, 8.0 MB/s 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T03:00:36.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:00:36.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T03:00:36.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:36.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:36.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:36.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:36.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:00:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:00:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:00:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:00:37.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T03:00:37.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:00:39.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:39.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:00:39.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:00:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:00:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:00:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:00:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:00:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T03:00:39.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:00:39.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T03:00:39.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:00:39.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:00:39.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:00:39.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:00:39.540 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:39.538+0000 7fa813e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:39.542 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:39.540+0000 7fa813e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:39.543 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:39.541+0000 7fa813e0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:00:39.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:39.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:40.057 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:40.055+0000 7fa813e0a440 -1 Falling back to public interface 2024-09-15T03:00:40.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:00:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:42.575 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:42.573+0000 7fa813e0a440 -1 osd.1 105 log_to_monitors true 2024-09-15T03:00:42.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:42.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:42.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:00:42.610 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:42.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:42.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:43.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:00:44.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:00:44.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:00:44.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:00:44.056 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:00:44.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:00:44.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 106 last_clean_interval [103,105) [v2:127.0.0.1:6810/219926,v1:127.0.0.1:6811/219926] [v2:127.0.0.1:6812/219926,v1:127.0.0.1:6813/219926] exists,up 14174469-0503-4543-a4e2-260af417188e 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:00:44.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:00:44.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:00:44.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:00:44.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:00:44.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:00:44.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:00:44.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:00:44.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:00:44.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:00:44.985 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:00:44.985 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:00:44.985 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:00:44.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:00:44.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:44.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:00:45.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803845 2024-09-15T03:00:45.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803845 2024-09-15T03:00:45.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845' 2024-09-15T03:00:45.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:45.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:00:45.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=472446402563 2024-09-15T03:00:45.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 472446402563 2024-09-15T03:00:45.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-472446402563' 2024-09-15T03:00:45.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:45.208 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:00:45.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378678 2024-09-15T03:00:45.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378678 2024-09-15T03:00:45.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-472446402563 2-81604378678' 2024-09-15T03:00:45.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:00:45.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:00:45.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=395136991245 2024-09-15T03:00:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 395136991245 2024-09-15T03:00:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-472446402563 2-81604378678 3-395136991245' 2024-09-15T03:00:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:45.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803845 2024-09-15T03:00:45.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:00:45.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803845 2024-09-15T03:00:45.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:45.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803845 2024-09-15T03:00:45.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803845' 2024-09-15T03:00:45.422 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803845 2024-09-15T03:00:45.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:45.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803843 -lt 25769803845 2024-09-15T03:00:45.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:00:46.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:00:46.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:00:47.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803845 -lt 25769803845 2024-09-15T03:00:47.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:47.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-472446402563 2024-09-15T03:00:47.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:47.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:00:47.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-472446402563 2024-09-15T03:00:47.140 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:47.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=472446402563 2024-09-15T03:00:47.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 472446402563' 2024-09-15T03:00:47.141 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 472446402563 2024-09-15T03:00:47.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:00:47.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 472446402563 -lt 472446402563 2024-09-15T03:00:47.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:47.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378678 2024-09-15T03:00:47.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:47.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:00:47.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378678 2024-09-15T03:00:47.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:47.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378678 2024-09-15T03:00:47.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378678' 2024-09-15T03:00:47.518 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378678 2024-09-15T03:00:47.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:00:47.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378678 -lt 81604378678 2024-09-15T03:00:47.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:00:47.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-395136991245 2024-09-15T03:00:47.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:00:47.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:00:47.871 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-395136991245 2024-09-15T03:00:47.872 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:00:47.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=395136991245 2024-09-15T03:00:47.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 395136991245' 2024-09-15T03:00:47.873 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 395136991245 2024-09-15T03:00:47.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:00:48.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 395136991245 -lt 395136991245 2024-09-15T03:00:48.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:00:48.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:48.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:00:48.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:00:48.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:00:48.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:00:48.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:00:48.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:00:48.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:00:48.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:00:49.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:00:49.044 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:00:49.044 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:00:49.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:00:49.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:00:49.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:00:49.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:00:49.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:00:49.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:00:50.187 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:00:50.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-0-256 fail 2024-09-15T03:00:50.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:00:50.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:00:50.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-0-256 2024-09-15T03:00:50.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T03:00:50.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T03:00:50.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87967-0-256 td/test-erasure-eio/COPY 2024-09-15T03:00:50.362 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-size-87967-0-256: (5) Input/output error 2024-09-15T03:00:50.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T03:00:50.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T03:00:50.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-15T03:00:50.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:00:50.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T03:00:50.949 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T03:00:50.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T03:00:51.518 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T03:00:51.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T03:00:51.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:00:51.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:51.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:00:51.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:51.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:51.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:51.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:51.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:51.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:51.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:51.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:51.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:51.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:51.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:51.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:51.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:51.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:51.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:51.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:51.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:51.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:51.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:00:51.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:51.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:51.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:51.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:00:51.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:51.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:51.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T03:00:51.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T03:00:51.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T03:00:51.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:00:51.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:00:51.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:00:51.676 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:00:51.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:00:51.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:00:51.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:00:51.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:00:51.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:00:51.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:51.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:00:51.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:00:51.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:00:51.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:00:51.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:00:51.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:00:51.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:00:51.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:51.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:51.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:00:51.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:00:51.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:00:51.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T03:00:51.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:00:51.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:51.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:51.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T03:00:51.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T03:00:51.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:52.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:00:52.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:00:52.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:00:52.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:52.353 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:52.354 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:52.354 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:00:52.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:52.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T03:00:52.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:00:52.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:00:52.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:00:52.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:00:52.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:00:52.431 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:00:52.431 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:52.431 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:52.431 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:52.431 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:52.431 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:52.432 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:00:52.432 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:00:52.432 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T03:00:52.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T03:00:52.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T03:00:52.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:00:52.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:00:52.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:00:52.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T03:00:52.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:00:53.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:00:53.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:53.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:53.336 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:53.337 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:53.337 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:53.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:53.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:53.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:00:53.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T03:00:53.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:00:53.612 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T03:00:53.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:00:54.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T03:00:54.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:00:54.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:00:54.625 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:00:54.625 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:54.625 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:54.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:00:54.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T03:00:54.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T03:00:54.693 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:00:54.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T03:00:54.702 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:00:52.126+0000 7f67926c9900 0 load: jerasure load: lrc 2024-09-15T03:00:54.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-09-15T03:00:54.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-09-15T03:00:54.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-09-15T03:00:54.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T03:00:54.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T03:00:54.708 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:00:54.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:00:54.710 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:00:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:00:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:00:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:00:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T03:00:54.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:00:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ea08c19-f9f4-4f43-a932-37a47f8fc28f 2024-09-15T03:00:54.712 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 8ea08c19-f9f4-4f43-a932-37a47f8fc28f 2024-09-15T03:00:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8ea08c19-f9f4-4f43-a932-37a47f8fc28f' 2024-09-15T03:00:54.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:00:54.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmTeZmU8wDKxAAgK0xcZoqEmQCD1keeucymA== 2024-09-15T03:00:54.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmTeZmU8wDKxAAgK0xcZoqEmQCD1keeucymA=="}' 2024-09-15T03:00:54.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ea08c19-f9f4-4f43-a932-37a47f8fc28f -i td/test-erasure-eio/0/new.json 2024-09-15T03:00:54.990 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:00:54.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T03:00:55.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmTeZmU8wDKxAAgK0xcZoqEmQCD1keeucymA== --osd-uuid 8ea08c19-f9f4-4f43-a932-37a47f8fc28f 2024-09-15T03:00:55.025 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:55.024+0000 7f09d151f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:55.036 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:55.035+0000 7f09d151f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:55.038 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:55.036+0000 7f09d151f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:00:55.038 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:55.037+0000 7f09d151f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:00:55.038 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:55.037+0000 7f09d151f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:00:55.038 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:55.037+0000 7f09d151f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:00:55.077 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:00:55.076+0000 7f09d151f440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T03:01:01.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T03:01:01.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:01:01.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:01:01.099 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T03:01:01.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:01:01.575 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T03:01:01.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:01:01.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:01:01.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:01:01.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:01:01.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:01:01.598 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:01.597+0000 7fa965a0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:01.600 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:01.599+0000 7fa965a0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:01.601 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:01.600+0000 7fa965a0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:01:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:01:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:01:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:01:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:01:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:01.927 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:01:01.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:01:01.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:01.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:02.118 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:02.117+0000 7fa965a0d440 -1 Falling back to public interface 2024-09-15T03:01:02.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:03.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:03.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:03.274 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:01:03.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:01:03.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:03.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:03.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:04.611 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:04.610+0000 7fa965a0d440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:01:04.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:04.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:04.674 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:01:04.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:01:04.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:04.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:05.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:06.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:06.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:06.188 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:01:06.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:01:06.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:06.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:06.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:07.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:07.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:07.609 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:01:07.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:01:07.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:07.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:07.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:08.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:08.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:08.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:01:08.989 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:01:08.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:08.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:09.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:10.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:10.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:10.472 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:01:10.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:01:10.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:10.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:10.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:11.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:11.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:11.854 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:01:11.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:01:11.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:11.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:12.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:13.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:13.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:13.309 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:01:13.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:01:13.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:13.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:13.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:14.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:14.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:14.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:01:14.697 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:01:14.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:14.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:15.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:16.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:16.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:16.096 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:01:16.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:01:16.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:16.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:16.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:17.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:17.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:17.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:01:17.918 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:01:17.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:17.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:18.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:19.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:19.744 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:01:19.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:19.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:01:19.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:19.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:20.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:21.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:21.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:21.557 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:01:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:01:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:21.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:22.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:23.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:23.361 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:01:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:01:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:23.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:23.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:24.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:24.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:24.929 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:01:24.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:01:24.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:24.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:25.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:26.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:26.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:26.416 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:01:26.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:01:26.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:26.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:26.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:27.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:27.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:27.874 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:01:27.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:01:27.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:27.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:01:28.241 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/222118,v1:127.0.0.1:6803/222118] [v2:127.0.0.1:6804/222118,v1:127.0.0.1:6805/222118] exists,up 8ea08c19-f9f4-4f43-a932-37a47f8fc28f 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:01:28.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:01:28.247 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:01:28.248 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:01:28.248 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:01:28.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:01:28.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:01:28.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:01:28.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:01:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:01:28.250 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:01:28.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cd50c0e9-fd9c-45ad-9b97-5914b0324865' 2024-09-15T03:01:28.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:01:28.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAITuZm8exuDxAAaCfSssBdP2EkKbMzhlEK6g== 2024-09-15T03:01:28.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAITuZm8exuDxAAaCfSssBdP2EkKbMzhlEK6g=="}' 2024-09-15T03:01:28.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd50c0e9-fd9c-45ad-9b97-5914b0324865 -i td/test-erasure-eio/1/new.json 2024-09-15T03:01:28.762 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:01:28.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T03:01:28.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAITuZm8exuDxAAaCfSssBdP2EkKbMzhlEK6g== --osd-uuid cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:01:28.798 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:28.797+0000 7f4dc4153440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:28.800 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:28.799+0000 7f4dc4153440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:28.801 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:28.800+0000 7f4dc4153440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:28.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:28.800+0000 7f4dc4153440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:01:28.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:28.800+0000 7f4dc4153440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:01:28.802 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:28.800+0000 7f4dc4153440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:01:28.877 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:28.876+0000 7f4dc4153440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T03:01:34.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T03:01:34.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:01:34.768 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T03:01:34.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:01:34.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:01:35.248 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T03:01:35.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:01:35.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:01:35.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:01:35.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:01:35.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:01:35.271 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:35.269+0000 7f006e727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:35.280 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:35.278+0000 7f006e727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:35.281 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:35.279+0000 7f006e727440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:01:35.661 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:01:35.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:35.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:35.818 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:35.816+0000 7f006e727440 -1 Falling back to public interface 2024-09-15T03:01:36.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:37.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:37.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:37.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:01:37.008 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:01:37.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:37.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:37.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:38.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:01:38.298+0000 7f006e727440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:01:38.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:38.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:38.357 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:01:38.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:01:38.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:38.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:38.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:39.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:39.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:39.868 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:01:39.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:01:39.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:39.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:40.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:41.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:41.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:41.355 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:01:41.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:01:41.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:41.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:41.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:42.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:42.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:42.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:01:42.824 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:01:42.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:42.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:43.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:44.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:44.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:44.322 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:01:44.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:01:44.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:44.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:44.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:45.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:45.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:45.706 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:01:45.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:01:45.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:45.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:46.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:47.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:47.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:47.185 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:01:47.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:01:47.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:47.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:47.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:48.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:48.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:48.660 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:01:48.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:01:48.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:48.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:49.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:50.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:50.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:50.174 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:01:50.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:01:50.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:50.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:50.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:51.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:51.746 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:01:51.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:51.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:01:51.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:51.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:52.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:53.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:53.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:53.925 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:01:53.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:01:53.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:53.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:54.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:55.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:55.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:55.434 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:01:55.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:01:55.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:55.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:55.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:56.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:56.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:56.981 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:01:56.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:01:56.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:56.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:57.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:01:58.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:01:58.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:01:58.905 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:01:58.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:01:58.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:01:58.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:01:59.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:00.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:00.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:00.490 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:02:00.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:02:00.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:00.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:02:01.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:02.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:02.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:02.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:02:02.052 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:02:02.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:02.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:02:02.486 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/223884,v1:127.0.0.1:6811/223884] [v2:127.0.0.1:6812/223884,v1:127.0.0.1:6813/223884] exists,up cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:02:02.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:02.491 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:02:02.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:02:02.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:02:02.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:02:02.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:02:02.493 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:02:02.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5bae0593-be68-41cf-a3d6-26caf057c493' 2024-09-15T03:02:02.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:02:02.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqTuZmtv8FHhAAmrn/I6y7lv9lKyZM1AhN8Q== 2024-09-15T03:02:02.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqTuZmtv8FHhAAmrn/I6y7lv9lKyZM1AhN8Q=="}' 2024-09-15T03:02:02.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5bae0593-be68-41cf-a3d6-26caf057c493 -i td/test-erasure-eio/2/new.json 2024-09-15T03:02:03.017 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:02:03.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T03:02:03.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqTuZmtv8FHhAAmrn/I6y7lv9lKyZM1AhN8Q== --osd-uuid 5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:02:03.054 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:03.053+0000 7f43ad84f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:03.075 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:03.075+0000 7f43ad84f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:03.076 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:03.075+0000 7f43ad84f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:03.077 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:03.076+0000 7f43ad84f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:02:03.077 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:03.076+0000 7f43ad84f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:02:03.077 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:03.076+0000 7f43ad84f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:02:03.170 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:03.169+0000 7f43ad84f440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T03:02:09.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T03:02:09.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:02:09.016 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T03:02:09.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:02:09.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:02:09.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:02:09.504 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:02:09.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:02:09.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:02:09.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:02:09.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:02:09.527 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:09.526+0000 7fde86d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:09.528 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:09.528+0000 7fde86d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:09.529 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:09.529+0000 7fde86d23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:09.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:02:09.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:02:09.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:02:09.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:02:09.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:02:09.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:09.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:02:09.856 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:02:09.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:09.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:10.064 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:10.063+0000 7fde86d23440 -1 Falling back to public interface 2024-09-15T03:02:10.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:11.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:11.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:11.209 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:02:11.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:02:11.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:11.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:11.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:12.531 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:12.530+0000 7fde86d23440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:02:12.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:12.568 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:02:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:02:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:12.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:13.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:14.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:14.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:14.320 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:02:14.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:02:14.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:14.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:14.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:15.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:15.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:15.693 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:02:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:02:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:16.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:17.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:17.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:17.185 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:02:17.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:02:17.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:17.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:17.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:18.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:18.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:18.697 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:02:18.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:02:18.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:18.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:19.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:20.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:20.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:20.151 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:02:20.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:02:20.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:20.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:20.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:21.619 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:02:21.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:02:21.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:21.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:22.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:23.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:23.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:23.005 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:02:23.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:02:23.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:23.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:23.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:24.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:24.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:24.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:02:24.478 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:02:24.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:24.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:25.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:26.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:26.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:26.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:02:26.076 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:02:26.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:26.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:26.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:27.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:27.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:27.920 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:02:27.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:02:27.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:27.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:28.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:29.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:29.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:29.831 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:02:29.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:02:29.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:29.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:30.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:31.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:31.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:31.626 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:02:31.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:02:31.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:31.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:32.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:33.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:33.757 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:02:33.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:02:33.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:33.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:34.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:35.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:35.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:35.550 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:02:35.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:02:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:35.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:36.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:36.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:36.960 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:02:36.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:02:36.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:36.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:02:37.413 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/225638,v1:127.0.0.1:6819/225638] [v2:127.0.0.1:6820/225638,v1:127.0.0.1:6821/225638] exists,up 5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:02:37.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:02:37.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:02:37.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:02:37.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:02:37.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T03:02:37.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:02:37.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:02:37.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aef6c6a1-a3c7-4579-a6b1-7305760b23ae 2024-09-15T03:02:37.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 aef6c6a1-a3c7-4579-a6b1-7305760b23ae' 2024-09-15T03:02:37.421 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 aef6c6a1-a3c7-4579-a6b1-7305760b23ae 2024-09-15T03:02:37.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:02:37.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNTuZmDlKpGRAAotVM5oN/q1Pe7D0VCKDLZg== 2024-09-15T03:02:37.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNTuZmDlKpGRAAotVM5oN/q1Pe7D0VCKDLZg=="}' 2024-09-15T03:02:37.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aef6c6a1-a3c7-4579-a6b1-7305760b23ae -i td/test-erasure-eio/3/new.json 2024-09-15T03:02:38.154 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:02:38.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T03:02:38.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNTuZmDlKpGRAAotVM5oN/q1Pe7D0VCKDLZg== --osd-uuid aef6c6a1-a3c7-4579-a6b1-7305760b23ae 2024-09-15T03:02:38.190 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:38.189+0000 7fbe5000d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:38.192 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:38.191+0000 7fbe5000d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:38.193 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:38.192+0000 7fbe5000d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:38.193 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:38.192+0000 7fbe5000d440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:02:38.193 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:38.192+0000 7fbe5000d440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:02:38.193 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:38.192+0000 7fbe5000d440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:02:38.261 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:38.260+0000 7fbe5000d440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T03:02:44.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T03:02:44.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:02:44.189 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T03:02:44.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T03:02:44.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:02:44.669 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T03:02:44.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T03:02:44.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:02:44.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:02:44.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:02:44.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:02:44.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:44.691+0000 7fe4f3d20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:44.694 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:44.693+0000 7fe4f3d20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:44.695 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:44.694+0000 7fe4f3d20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:02:45.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T03:02:45.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:02:45.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T03:02:45.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:02:45.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:02:45.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:45.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:02:45.040 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:02:45.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:45.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:45.209 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:45.208+0000 7fe4f3d20440 -1 Falling back to public interface 2024-09-15T03:02:45.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:46.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:46.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:46.410 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:02:46.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:02:46.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:46.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:46.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:47.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:47.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:47.767 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:02:47.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:02:47.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:47.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:47.807 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:02:47.806+0000 7fe4f3d20440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:02:48.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:49.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:49.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:49.142 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:02:49.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:02:49.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:49.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:49.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:50.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:50.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:50.733 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:02:50.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:02:50.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:50.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:51.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:52.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:52.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:52.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:02:52.140 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:02:52.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:52.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:52.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:53.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:53.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:53.636 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:02:53.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:02:53.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:53.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:54.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:55.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:55.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:55.003 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:02:55.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:02:55.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:55.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:55.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:56.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:56.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:56.468 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:02:56.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:02:56.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:56.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:57.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:57.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:57.879 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:02:57.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:02:57.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:57.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:58.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:02:59.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:02:59.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:02:59.290 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:02:59.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:02:59.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:02:59.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:02:59.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:00.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:00.803 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:03:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:03:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:01.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:02.262 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:03:02.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:03:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:02.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:03.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:03.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:03.875 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:03:03.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:03:03.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:03.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:04.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:05.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:05.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:05.626 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:03:05.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:03:05.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:05.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:06.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:07.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:07.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:07.285 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:03:07.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:03:07.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:07.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:08.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:09.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:09.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:09.088 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:03:09.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:03:09.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:09.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:09.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:10.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:10.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:10.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:03:10.928 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:03:10.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:10.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:11.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:12.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:12.377 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T03:03:12.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:12.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:03:12.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:12.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:03:12.800 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/227392,v1:127.0.0.1:6827/227392] [v2:127.0.0.1:6828/227392,v1:127.0.0.1:6829/227392] exists,up aef6c6a1-a3c7-4579-a6b1-7305760b23ae 2024-09-15T03:03:12.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T03:03:12.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T03:03:12.877 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:03:12.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T03:03:12.887 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:01:02.121+0000 7fa965a0d440 0 load: jerasure load: lrc 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T03:03:12.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T03:03:12.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T03:03:13.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T03:03:13.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T03:03:14.852 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T03:03:14.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:03:15.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T03:03:15.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:03:15.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:03:15.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:03:15.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:03:15.866 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:03:15.866 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:03:15.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:03:15.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:03:15.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:03:15.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:03:15.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:03:15.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:03:15.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:03:15.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:03:15.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:03:16.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:03:16.392 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:16.392 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:03:16.392 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:03:16.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:03:16.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:16.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:03:16.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803803 2024-09-15T03:03:16.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803803 2024-09-15T03:03:16.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803' 2024-09-15T03:03:16.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:16.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:03:16.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574869 2024-09-15T03:03:16.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574869 2024-09-15T03:03:16.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574869' 2024-09-15T03:03:16.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:16.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:03:16.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378639 2024-09-15T03:03:16.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378639 2024-09-15T03:03:16.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574869 2-81604378639' 2024-09-15T03:03:16.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:16.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:03:16.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084295 2024-09-15T03:03:16.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084295 2024-09-15T03:03:16.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574869 2-81604378639 3-120259084295' 2024-09-15T03:03:16.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:16.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803803 2024-09-15T03:03:16.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:16.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:03:16.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803803 2024-09-15T03:03:16.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:16.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803803 2024-09-15T03:03:16.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-09-15T03:03:16.815 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803803 2024-09-15T03:03:16.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:03:17.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803803 -lt 25769803803 2024-09-15T03:03:17.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:17.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574869 2024-09-15T03:03:17.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:17.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:03:17.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574869 2024-09-15T03:03:17.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:17.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574869 2024-09-15T03:03:17.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574869' 2024-09-15T03:03:17.164 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574869 2024-09-15T03:03:17.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:03:17.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574870 -lt 55834574869 2024-09-15T03:03:17.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:17.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378639 2024-09-15T03:03:17.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:17.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:03:17.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378639 2024-09-15T03:03:17.521 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:17.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378639 2024-09-15T03:03:17.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2024-09-15T03:03:17.523 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378639 2024-09-15T03:03:17.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:03:17.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378639 -lt 81604378639 2024-09-15T03:03:17.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:17.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084295 2024-09-15T03:03:17.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:17.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:03:17.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084295 2024-09-15T03:03:17.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:17.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084295 2024-09-15T03:03:17.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084295' 2024-09-15T03:03:17.877 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084295 2024-09-15T03:03:17.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:03:18.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084295 -lt 120259084295 2024-09-15T03:03:18.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:03:18.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:18.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:18.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:03:18.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:03:18.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:03:18.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:03:18.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:03:18.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:03:18.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:03:18.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:03:19.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:03:19.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:03:19.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:19.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:19.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:03:19.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:03:19.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:03:19.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-09-15T03:03:19.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-87967-1-10 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87967-1-10 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:03:19.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87967-1-10 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:03:19.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87967-1-10 td/test-erasure-eio/ORIGINAL 2024-09-15T03:03:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-87967-1-10 td/test-erasure-eio 1 10 set 2024-09-15T03:03:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-1-10 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-1-10 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:03:19.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-1-10 2024-09-15T03:03:19.668 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-1-10 2024-09-15T03:03:19.668 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-15T03:03:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T03:03:20.760 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T03:03:20.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T03:03:20.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-15T03:03:20.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:10 bytes copied, 4.9821e-05 s, 201 kB/s 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:03:20.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T03:03:20.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:03:20.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T03:03:20.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:03:20.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:03:20.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:03:20.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:03:20.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:03:21.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:03:21.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:21.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:03:21.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:03:21.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:03:21.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:03:21.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T03:03:21.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:03:23.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:03:23.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:03:23.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:03:23.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:03:23.662 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T03:03:23.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:03:23.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T03:03:23.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:03:23.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:03:23.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:03:23.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:03:23.685 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:23.683+0000 7f6c9d123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:23.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:23.685+0000 7f6c9d123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:23.692 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:23.686+0000 7f6c9d123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:24.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:03:24.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:03:24.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:03:24.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:03:24.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:03:24.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:24.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:03:24.022 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:03:24.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:24.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:24.206 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:24.204+0000 7f6c9d123440 -1 Falling back to public interface 2024-09-15T03:03:24.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:25.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:25.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:25.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:03:25.383 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:25.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:25.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:25.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:26.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:26.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:26.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:03:26.750 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:03:26.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:26.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:26.773 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:26.772+0000 7f6c9d123440 -1 osd.1 36 log_to_monitors true 2024-09-15T03:03:27.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:28.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:28.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:28.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:03:28.223 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:03:28.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:28.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:28.686 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [13,36) [v2:127.0.0.1:6810/229221,v1:127.0.0.1:6811/229221] [v2:127.0.0.1:6812/229221,v1:127.0.0.1:6813/229221] exists,up cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:03:28.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:03:28.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:03:28.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:03:28.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:03:28.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:03:28.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:03:28.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:03:28.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:03:28.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:03:28.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:03:28.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:03:28.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:03:28.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:03:28.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:03:28.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:03:28.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:03:28.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:03:28.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:03:28.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:03:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:03:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:03:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:03:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:03:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:29.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:03:29.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803808 2024-09-15T03:03:29.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803808 2024-09-15T03:03:29.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808' 2024-09-15T03:03:29.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:29.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:03:29.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=184683593731 2024-09-15T03:03:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 184683593731 2024-09-15T03:03:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-184683593731' 2024-09-15T03:03:29.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:29.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:03:29.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378644 2024-09-15T03:03:29.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378644 2024-09-15T03:03:29.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-184683593731 2-81604378644' 2024-09-15T03:03:29.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:29.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:03:29.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084300 2024-09-15T03:03:29.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084300 2024-09-15T03:03:29.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-184683593731 2-81604378644 3-120259084300' 2024-09-15T03:03:29.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:29.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803808 2024-09-15T03:03:29.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:29.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:03:29.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803808 2024-09-15T03:03:29.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:29.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803808 2024-09-15T03:03:29.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-15T03:03:29.720 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803808 2024-09-15T03:03:29.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:03:30.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803806 -lt 25769803808 2024-09-15T03:03:30.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:03:31.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:03:31.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:03:31.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-15T03:03:31.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:31.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-184683593731 2024-09-15T03:03:31.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:31.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:03:31.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-184683593731 2024-09-15T03:03:31.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:31.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=184683593731 2024-09-15T03:03:31.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-09-15T03:03:31.473 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 184683593731 2024-09-15T03:03:31.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:03:31.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 184683593731 -lt 184683593731 2024-09-15T03:03:31.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:31.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378644 2024-09-15T03:03:31.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:31.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:03:31.824 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378644 2024-09-15T03:03:31.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:31.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378644 2024-09-15T03:03:31.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-09-15T03:03:31.826 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 81604378644 2024-09-15T03:03:31.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:03:32.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378644 -lt 81604378644 2024-09-15T03:03:32.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:32.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084300 2024-09-15T03:03:32.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:32.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:03:32.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084300 2024-09-15T03:03:32.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:32.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084300 2024-09-15T03:03:32.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084300' 2024-09-15T03:03:32.182 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084300 2024-09-15T03:03:32.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:03:32.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084300 -lt 120259084300 2024-09-15T03:03:32.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:03:32.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:32.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:33.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:03:33.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:03:33.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:03:33.042 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:03:33.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:03:33.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:03:33.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:03:33.043 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:03:33.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:03:33.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:03:33.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:33.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:33.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:03:33.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:03:33.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:03:33.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:03:33.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:03:34.577 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:03:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-1-10 2024-09-15T03:03:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:03:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:03:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-1-10 2024-09-15T03:03:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:03:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:03:34.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87967-1-10 td/test-erasure-eio/COPY 2024-09-15T03:03:34.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:03:34.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:03:34.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-87967-1-10 td/test-erasure-eio 2 10 set 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-1-10 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T03:03:34.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-1-10 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-1-10 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-1-10 2024-09-15T03:03:34.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-15T03:03:35.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T03:03:35.875 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T03:03:35.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T03:03:35.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-15T03:03:35.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:10 bytes copied, 8.4185e-05 s, 119 kB/s 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:35.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:03:35.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:03:36.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:03:36.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:36.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:03:36.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:03:36.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-15T03:03:36.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:03:36.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T03:03:36.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87967-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:38.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:03:38.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:03:38.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:03:38.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:03:38.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:03:38.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:03:38.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:03:38.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:03:38.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.2 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:03:38.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:03:38.759 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:38.758+0000 7f4fb0986440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:38.761 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:38.760+0000 7f4fb0986440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:38.762 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:38.761+0000 7f4fb0986440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:39.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:03:39.289 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:39.288+0000 7f4fb0986440 -1 Falling back to public interface 2024-09-15T03:03:39.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:40.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:40.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:40.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:03:40.454 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:40.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:40.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:03:40.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:03:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:03:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:03:41.924 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:41.923+0000 7f4fb0986440 -1 osd.2 47 log_to_monitors true 2024-09-15T03:03:42.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:03:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:03:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:43.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:03:43.608 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 up in weight 1 up_from 53 up_thru 39 down_at 50 last_clean_interval [19,47) [v2:127.0.0.1:6818/230673,v1:127.0.0.1:6819/230673] [v2:127.0.0.1:6820/230673,v1:127.0.0.1:6821/230673] exists,up 5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:03:43.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:03:43.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:03:43.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:03:43.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:03:43.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:03:43.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:03:43.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:03:43.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:03:43.610 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:03:43.610 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:03:43.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:03:43.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:03:43.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:03:43.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:03:43.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:03:43.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:03:43.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:03:43.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:03:43.699 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:03:44.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:03:44.162 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:44.162 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:03:44.162 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:03:44.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:03:44.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:44.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:03:44.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803813 2024-09-15T03:03:44.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803813 2024-09-15T03:03:44.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813' 2024-09-15T03:03:44.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:44.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:03:44.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=184683593736 2024-09-15T03:03:44.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 184683593736 2024-09-15T03:03:44.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-184683593736' 2024-09-15T03:03:44.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:44.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:03:44.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=227633266691 2024-09-15T03:03:44.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 227633266691 2024-09-15T03:03:44.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-184683593736 2-227633266691' 2024-09-15T03:03:44.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:44.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:03:44.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084305 2024-09-15T03:03:44.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084305 2024-09-15T03:03:44.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-184683593736 2-227633266691 3-120259084305' 2024-09-15T03:03:44.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:44.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803813 2024-09-15T03:03:44.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:44.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:03:44.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803813 2024-09-15T03:03:44.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:44.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803813 2024-09-15T03:03:44.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-15T03:03:44.613 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803813 2024-09-15T03:03:44.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:03:44.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803813 2024-09-15T03:03:44.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:03:45.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:03:45.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:03:46.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803813 -lt 25769803813 2024-09-15T03:03:46.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:46.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-184683593736 2024-09-15T03:03:46.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:46.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:03:46.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-184683593736 2024-09-15T03:03:46.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:46.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=184683593736 2024-09-15T03:03:46.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 184683593736' 2024-09-15T03:03:46.320 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 184683593736 2024-09-15T03:03:46.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:03:46.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 184683593736 -lt 184683593736 2024-09-15T03:03:46.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:46.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-227633266691 2024-09-15T03:03:46.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:46.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:03:46.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-227633266691 2024-09-15T03:03:46.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:46.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=227633266691 2024-09-15T03:03:46.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 227633266691' 2024-09-15T03:03:46.680 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 227633266691 2024-09-15T03:03:46.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:03:47.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 227633266691 -lt 227633266691 2024-09-15T03:03:47.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:03:47.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084305 2024-09-15T03:03:47.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:03:47.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:03:47.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084305 2024-09-15T03:03:47.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:03:47.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084305 2024-09-15T03:03:47.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084305' 2024-09-15T03:03:47.054 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084305 2024-09-15T03:03:47.054 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:03:47.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084305 -lt 120259084305 2024-09-15T03:03:47.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:03:47.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:47.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:03:47.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:03:48.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:03:48.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:03:48.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:03:48.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:03:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:03:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:03:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:03:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:03:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:03:49.481 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:03:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-1-10 fail 2024-09-15T03:03:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:03:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:03:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-1-10 2024-09-15T03:03:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T03:03:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T03:03:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87967-1-10 td/test-erasure-eio/COPY 2024-09-15T03:03:49.622 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-size-87967-1-10: (5) Input/output error 2024-09-15T03:03:49.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T03:03:49.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T03:03:49.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:324: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2024-09-15T03:03:49.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T03:03:49.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-15T03:03:49.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-87967-1-0 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87967-1-0 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87967-1-0 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:49.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:03:49.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:49.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:03:49.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:03:49.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:03:49.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87967-1-0 td/test-erasure-eio/ORIGINAL 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-87967-1-0 td/test-erasure-eio 1 0 set 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-1-0 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-15T03:03:49.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T03:03:49.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T03:03:49.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T03:03:49.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-1-0 2024-09-15T03:03:49.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:03:49.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-1-0 2024-09-15T03:03:49.787 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-1-0 2024-09-15T03:03:49.787 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:03:50.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:03:50.202 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:50.203 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:03:50.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:03:50.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T03:03:50.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T03:03:50.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-15T03:03:50.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T03:03:50.938 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T03:03:50.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T03:03:50.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-15T03:03:50.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:03:50.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T03:03:50.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:03:50.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T03:03:50.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:03:50.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:03:50.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:03:50.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:03:50.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:03:52.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:03:52.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:52.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:03:52.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:03:52.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:03:52.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:03:52.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T03:03:52.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:03:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T03:03:54.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:03:54.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:03:54.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:03:54.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:03:54.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:03:54.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:03:54.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:03:54.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:03:54.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:03:54.164 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:54.163+0000 7fd0f780e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:54.166 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:54.165+0000 7fd0f780e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:54.168 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:54.167+0000 7fd0f780e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:03:54.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:54.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:54.684 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:54.683+0000 7fd0f780e440 -1 Falling back to public interface 2024-09-15T03:03:54.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:55.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:55.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:55.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:03:55.855 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:55.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:55.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:56.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:57.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:57.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:57.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:03:57.209 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:03:57.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:57.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:57.360 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:03:57.359+0000 7fd0f780e440 -1 osd.1 56 log_to_monitors true 2024-09-15T03:03:57.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:03:58.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:03:58.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:03:58.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:03:58.626 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:03:58.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:03:58.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:03:59.217 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [43,56) [v2:127.0.0.1:6810/232144,v1:127.0.0.1:6811/232144] [v2:127.0.0.1:6812/232144,v1:127.0.0.1:6813/232144] exists,up cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:03:59.218 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:03:59.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:03:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:03:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:03:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:03:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:03:59.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:03:59.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:03:59.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:03:59.693 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:03:59.694 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:03:59.694 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:03:59.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:03:59.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:59.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:03:59.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803818 2024-09-15T03:03:59.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803818 2024-09-15T03:03:59.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818' 2024-09-15T03:03:59.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:59.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:03:59.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=266287972355 2024-09-15T03:03:59.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 266287972355 2024-09-15T03:03:59.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818 1-266287972355' 2024-09-15T03:03:59.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:03:59.916 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:04:00.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=227633266696 2024-09-15T03:04:00.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 227633266696 2024-09-15T03:04:00.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818 1-266287972355 2-227633266696' 2024-09-15T03:04:00.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:00.021 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:04:00.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084310 2024-09-15T03:04:00.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084310 2024-09-15T03:04:00.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803818 1-266287972355 2-227633266696 3-120259084310' 2024-09-15T03:04:00.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:00.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803818 2024-09-15T03:04:00.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:00.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:04:00.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803818 2024-09-15T03:04:00.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:00.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803818 2024-09-15T03:04:00.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-09-15T03:04:00.129 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803818 2024-09-15T03:04:00.129 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:00.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803816 -lt 25769803818 2024-09-15T03:04:00.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:04:01.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:04:01.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:01.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803818 -lt 25769803818 2024-09-15T03:04:01.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:01.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-266287972355 2024-09-15T03:04:01.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:01.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:04:01.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-266287972355 2024-09-15T03:04:01.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:01.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=266287972355 2024-09-15T03:04:01.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 266287972355' 2024-09-15T03:04:01.877 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 266287972355 2024-09-15T03:04:01.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:04:02.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 266287972355 -lt 266287972355 2024-09-15T03:04:02.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:02.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-227633266696 2024-09-15T03:04:02.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:02.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:04:02.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-227633266696 2024-09-15T03:04:02.234 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:02.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=227633266696 2024-09-15T03:04:02.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 227633266696' 2024-09-15T03:04:02.236 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 227633266696 2024-09-15T03:04:02.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:04:02.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 227633266696 -lt 227633266696 2024-09-15T03:04:02.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:02.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084310 2024-09-15T03:04:02.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:02.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:04:02.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084310 2024-09-15T03:04:02.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:02.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084310 2024-09-15T03:04:02.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084310' 2024-09-15T03:04:02.590 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084310 2024-09-15T03:04:02.590 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:04:02.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084310 -lt 120259084310 2024-09-15T03:04:02.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:04:02.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:02.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:03.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:03.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:04:03.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:04:03.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:03.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:04.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:04:04.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:04:04.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:04:04.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:04:04.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:04:04.863 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:04:04.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-1-0 2024-09-15T03:04:04.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:04:04.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:04:04.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-1-0 2024-09-15T03:04:04.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:04:04.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:04:04.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87967-1-0 td/test-erasure-eio/COPY 2024-09-15T03:04:04.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:04:04.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:04:05.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-15T03:04:05.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-87967-1-0 td/test-erasure-eio 2 0 set 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-1-0 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-1-0 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:04:05.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-1-0 2024-09-15T03:04:05.018 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-1-0 2024-09-15T03:04:05.018 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:04:05.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:04:05.369 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:05.369 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:04:05.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:04:05.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T03:04:05.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T03:04:05.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-15T03:04:05.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T03:04:06.196 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T03:04:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-15T03:04:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-15T03:04:06.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-15T03:04:06.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:06.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:04:06.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:04:06.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:06.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:06.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:06.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:06.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:06.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:04:06.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:04:06.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-15T03:04:06.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:04:06.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T03:04:06.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87967-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:08.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T03:04:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:04:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:04:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:04:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:04:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:04:08.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:04:08.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:04:08.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:04:08.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:04:08.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:04:08.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:04:08.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:04:08.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:04:08.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:04:08.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:04:08.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:08.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:08.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:08.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:08.749 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:04:08.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:08.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:04:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:04:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:04:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:04:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:04:08.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:04:08.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:04:08.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:04:08.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:04:08.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:04:08.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:04:08.752 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.2 2024-09-15T03:04:08.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:04:08.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T03:04:08.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:04:08.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:04:08.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:04:08.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:04:08.768 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:08.766+0000 7f1634c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:08.769 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:08.768+0000 7f1634c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:08.770 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:08.770+0000 7f1634c0a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:09.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:04:09.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:04:09.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:04:09.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:04:09.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:04:09.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:09.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:04:09.122 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:04:09.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:09.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:09.288 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:09.287+0000 7f1634c0a440 -1 Falling back to public interface 2024-09-15T03:04:09.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:04:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:10.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:10.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:11.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:04:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:11.878 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:11.877+0000 7f1634c0a440 -1 osd.2 66 log_to_monitors true 2024-09-15T03:04:12.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:04:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:04:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:13.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:13.941 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 up in weight 1 up_from 72 up_thru 59 down_at 69 last_clean_interval [53,66) [v2:127.0.0.1:6818/233596,v1:127.0.0.1:6819/233596] [v2:127.0.0.1:6820/233596,v1:127.0.0.1:6821/233596] exists,up 5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:04:13.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:04:13.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:04:13.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:04:13.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:04:13.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:04:13.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:04:13.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:04:13.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:04:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:04:13.943 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:04:13.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:04:13.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:04:13.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:04:14.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:04:14.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:04:14.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:04:14.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:04:14.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:04:14.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:04:14.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:04:14.502 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:14.502 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:14.502 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:04:14.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:04:14.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:14.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:14.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803823 2024-09-15T03:04:14.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803823 2024-09-15T03:04:14.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803823' 2024-09-15T03:04:14.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:14.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:04:14.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=266287972360 2024-09-15T03:04:14.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 266287972360 2024-09-15T03:04:14.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803823 1-266287972360' 2024-09-15T03:04:14.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:14.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:04:14.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645315 2024-09-15T03:04:14.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645315 2024-09-15T03:04:14.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803823 1-266287972360 2-309237645315' 2024-09-15T03:04:14.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:14.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:04:14.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084315 2024-09-15T03:04:14.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084315 2024-09-15T03:04:14.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803823 1-266287972360 2-309237645315 3-120259084315' 2024-09-15T03:04:14.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:14.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803823 2024-09-15T03:04:14.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:14.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:04:14.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803823 2024-09-15T03:04:14.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803823 2024-09-15T03:04:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-09-15T03:04:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803823 2024-09-15T03:04:14.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:15.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803823 -lt 25769803823 2024-09-15T03:04:15.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:15.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-266287972360 2024-09-15T03:04:15.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:15.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:04:15.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-266287972360 2024-09-15T03:04:15.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:15.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=266287972360 2024-09-15T03:04:15.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 266287972360' 2024-09-15T03:04:15.290 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 266287972360 2024-09-15T03:04:15.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:04:15.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 266287972360 -lt 266287972360 2024-09-15T03:04:15.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:15.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-309237645315 2024-09-15T03:04:15.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:15.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:04:15.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-309237645315 2024-09-15T03:04:15.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:15.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645315 2024-09-15T03:04:15.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 309237645315' 2024-09-15T03:04:15.666 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 309237645315 2024-09-15T03:04:15.666 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:04:16.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645315 -lt 309237645315 2024-09-15T03:04:16.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:16.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084315 2024-09-15T03:04:16.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:16.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:04:16.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084315 2024-09-15T03:04:16.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:16.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084315 2024-09-15T03:04:16.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084315' 2024-09-15T03:04:16.061 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084315 2024-09-15T03:04:16.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:04:16.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084313 -lt 120259084315 2024-09-15T03:04:16.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:04:17.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:04:17.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:04:17.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084315 -lt 120259084315 2024-09-15T03:04:17.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:04:17.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:17.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:18.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:18.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:04:18.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:04:18.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:18.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:19.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:04:19.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:04:19.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:04:19.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:04:19.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:04:19.764 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:04:19.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-1-0 fail 2024-09-15T03:04:19.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:04:19.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:04:19.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-1-0 2024-09-15T03:04:19.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T03:04:19.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T03:04:19.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87967-1-0 td/test-erasure-eio/COPY 2024-09-15T03:04:19.959 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-size-87967-1-0: (5) Input/output error 2024-09-15T03:04:19.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T03:04:19.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T03:04:19.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-87967-1-256 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-87967-1-256 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:04:19.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-87967-1-256 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:04:19.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-87967-1-256 td/test-erasure-eio/ORIGINAL 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-87967-1-256 td/test-erasure-eio 1 256 add 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-1-256 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-15T03:04:20.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T03:04:20.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-15T03:04:20.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T03:04:20.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-1-256 2024-09-15T03:04:20.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:04:20.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-1-256 2024-09-15T03:04:20.188 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-1-256 2024-09-15T03:04:20.189 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:04:20.627 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:04:20.627 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:20.627 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:04:20.628 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:04:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T03:04:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T03:04:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-15T03:04:20.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T03:04:21.363 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T03:04:21.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-15T03:04:21.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:21.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:04:21.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:21.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T03:04:21.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:23.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T03:04:23.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:04:23.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:04:23.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:04:23.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:04:23.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:04:23.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:04:23.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:04:23.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:04:23.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:04:23.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:04:23.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:04:23.941 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T03:04:23.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:04:23.941 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T03:04:23.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:04:23.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:04:23.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:04:23.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:04:23.963 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:23.961+0000 7fc92d91f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:23.965 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:23.964+0000 7fc92d91f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:23.967 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:23.965+0000 7fc92d91f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:24.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:04:24.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:04:24.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:04:24.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:04:24.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:04:24.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:24.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:04:24.298 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:04:24.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:24.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:24.489 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:24.487+0000 7fc92d91f440 -1 Falling back to public interface 2024-09-15T03:04:24.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:25.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:25.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:25.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:04:25.674 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:25.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:25.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:26.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:26.974 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:26.972+0000 7fc92d91f440 -1 osd.1 76 log_to_monitors true 2024-09-15T03:04:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:04:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:27.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:27.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:27.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:28.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:28.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:28.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:04:28.449 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:04:28.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:28.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [62,76) [v2:127.0.0.1:6810/235066,v1:127.0.0.1:6811/235066] [v2:127.0.0.1:6812/235066,v1:127.0.0.1:6813/235066] exists,up cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:04:28.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:04:28.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:04:28.982 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:04:28.982 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:04:28.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:04:28.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:04:28.983 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:04:29.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:04:29.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:04:29.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:04:29.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:04:29.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:04:29.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:04:29.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:04:29.500 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:29.500 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:29.501 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:04:29.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:04:29.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:29.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:29.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803828 2024-09-15T03:04:29.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803828 2024-09-15T03:04:29.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828' 2024-09-15T03:04:29.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:29.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:04:29.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=352187318275 2024-09-15T03:04:29.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 352187318275 2024-09-15T03:04:29.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-352187318275' 2024-09-15T03:04:29.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:29.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:04:29.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645320 2024-09-15T03:04:29.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645320 2024-09-15T03:04:29.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-352187318275 2-309237645320' 2024-09-15T03:04:29.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:29.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:04:29.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084320 2024-09-15T03:04:29.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084320 2024-09-15T03:04:29.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-352187318275 2-309237645320 3-120259084320' 2024-09-15T03:04:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:29.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803828 2024-09-15T03:04:29.929 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:29.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:04:29.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803828 2024-09-15T03:04:29.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803828 2024-09-15T03:04:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-09-15T03:04:29.931 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803828 2024-09-15T03:04:29.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:30.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803826 -lt 25769803828 2024-09-15T03:04:30.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:04:31.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:04:31.276 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:31.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803828 -lt 25769803828 2024-09-15T03:04:31.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:31.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-352187318275 2024-09-15T03:04:31.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:31.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:04:31.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-352187318275 2024-09-15T03:04:31.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:31.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=352187318275 2024-09-15T03:04:31.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2024-09-15T03:04:31.645 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 352187318275 2024-09-15T03:04:31.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:04:31.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 352187318275 -lt 352187318275 2024-09-15T03:04:32.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:32.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-309237645320 2024-09-15T03:04:32.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:32.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:04:32.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-309237645320 2024-09-15T03:04:32.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:32.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645320 2024-09-15T03:04:32.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 309237645320' 2024-09-15T03:04:32.003 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 309237645320 2024-09-15T03:04:32.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:04:32.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645320 -lt 309237645320 2024-09-15T03:04:32.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:32.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084320 2024-09-15T03:04:32.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:32.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:04:32.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084320 2024-09-15T03:04:32.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:32.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084320 2024-09-15T03:04:32.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084320' 2024-09-15T03:04:32.354 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084320 2024-09-15T03:04:32.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:04:32.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084320 -lt 120259084320 2024-09-15T03:04:32.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:04:32.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:32.706 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:33.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:04:33.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:04:33.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:04:33.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:33.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:33.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:33.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:33.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:33.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:04:33.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:04:33.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:33.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:33.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:04:33.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:04:33.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:04:33.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T03:04:33.984 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T03:04:33.984 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T03:04:33.984 INFO:tasks.workunit.client.0.smithi130.stderr:256 bytes copied, 3.136e-05 s, 8.2 MB/s 2024-09-15T03:04:33.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:04:33.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T03:04:33.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:33.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:33.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:33.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:33.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T03:04:34.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:36.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T03:04:36.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:04:36.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:04:36.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:04:36.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:36.807 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:04:36.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:04:36.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:04:36.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:04:36.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:04:36.810 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T03:04:36.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:04:36.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T03:04:36.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:04:36.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:04:36.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:04:36.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:04:36.832 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:36.831+0000 7f0564921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:36.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:36.833+0000 7f0564921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:36.836 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:36.835+0000 7f0564921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:37.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:04:37.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:37.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:37.350 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:37.349+0000 7f0564921440 -1 Falling back to public interface 2024-09-15T03:04:37.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:38.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:38.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:38.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:04:38.528 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:38.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:38.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:38.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:39.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:39.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:39.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:04:39.888 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:39.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:39.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:39.945 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:39.944+0000 7f0564921440 -1 osd.1 84 log_to_monitors true 2024-09-15T03:04:40.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:41.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:41.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:41.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:04:41.426 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:04:41.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:41.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:04:41.910 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [82,84) [v2:127.0.0.1:6810/236422,v1:127.0.0.1:6811/236422] [v2:127.0.0.1:6812/236422,v1:127.0.0.1:6813/236422] exists,up cd50c0e9-fd9c-45ad-9b97-5914b0324865 2024-09-15T03:04:41.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:04:41.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:04:41.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:04:41.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:04:41.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:04:41.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:04:41.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:04:41.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:04:41.912 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:04:41.912 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:04:41.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:04:41.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:04:41.912 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:04:42.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:04:42.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:04:42.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:04:42.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:04:42.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:04:42.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:04:42.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:04:42.344 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:42.344 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:42.344 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:04:42.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:04:42.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:42.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803832 2024-09-15T03:04:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803832 2024-09-15T03:04:42.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832' 2024-09-15T03:04:42.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:42.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:04:42.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=382252089347 2024-09-15T03:04:42.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 382252089347 2024-09-15T03:04:42.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-382252089347' 2024-09-15T03:04:42.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:42.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:04:42.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=309237645325 2024-09-15T03:04:42.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 309237645325 2024-09-15T03:04:42.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-382252089347 2-309237645325' 2024-09-15T03:04:42.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:42.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:04:42.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084325 2024-09-15T03:04:42.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084325 2024-09-15T03:04:42.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-382252089347 2-309237645325 3-120259084325' 2024-09-15T03:04:42.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:42.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803832 2024-09-15T03:04:42.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:42.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:04:42.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803832 2024-09-15T03:04:42.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:42.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803832 2024-09-15T03:04:42.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-15T03:04:42.789 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803832 2024-09-15T03:04:42.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:43.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803832 -lt 25769803832 2024-09-15T03:04:43.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:43.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-382252089347 2024-09-15T03:04:43.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:43.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:04:43.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-382252089347 2024-09-15T03:04:43.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:43.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=382252089347 2024-09-15T03:04:43.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 382252089347' 2024-09-15T03:04:43.149 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 382252089347 2024-09-15T03:04:43.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:04:43.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 382252089347 -lt 382252089347 2024-09-15T03:04:43.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:43.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-309237645325 2024-09-15T03:04:43.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:43.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:04:43.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-309237645325 2024-09-15T03:04:43.501 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:43.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=309237645325 2024-09-15T03:04:43.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 309237645325' 2024-09-15T03:04:43.502 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 309237645325 2024-09-15T03:04:43.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:04:43.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 309237645325 -lt 309237645325 2024-09-15T03:04:43.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:43.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084325 2024-09-15T03:04:43.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:43.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:04:43.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084325 2024-09-15T03:04:43.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:43.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084325 2024-09-15T03:04:43.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084325' 2024-09-15T03:04:43.860 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084325 2024-09-15T03:04:43.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:04:44.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084325 -lt 120259084325 2024-09-15T03:04:44.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:04:44.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:44.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:44.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:04:44.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:04:44.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:04:44.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:44.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:44.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:44.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:44.684 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:45.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:04:45.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:04:45.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:45.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:45.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:04:45.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:04:45.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:04:45.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:04:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:04:46.160 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:04:46.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-1-256 2024-09-15T03:04:46.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:04:46.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:04:46.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-1-256 2024-09-15T03:04:46.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:04:46.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:04:46.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-87967-1-256 td/test-erasure-eio/COPY 2024-09-15T03:04:46.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:04:46.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:04:46.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-15T03:04:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-15T03:04:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-87967-1-256 td/test-erasure-eio 2 256 add 2024-09-15T03:04:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-87967-1-256 2024-09-15T03:04:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-15T03:04:46.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-87967-1-256 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-87967-1-256 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-87967-1-256 2024-09-15T03:04:46.333 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:04:46.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:04:46.701 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:46.702 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:04:46.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:04:46.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-09-15T03:04:46.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-15T03:04:46.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-15T03:04:46.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-15T03:04:47.399 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-15T03:04:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-15T03:04:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:04:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:04:47.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:04:47.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T03:04:47.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87967-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:04:49.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T03:04:49.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:04:49.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:04:49.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:04:49.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:04:49.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:04:49.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:04:49.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:04:49.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:04:49.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:04:49.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.2 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:04:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:04:49.970 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:49.968+0000 7f08c4a15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:49.972 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:49.971+0000 7f08c4a15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:49.973 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:49.972+0000 7f08c4a15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:04:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:04:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:04:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:04:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:04:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:04:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:50.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:04:50.305 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:04:50.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:50.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:50.490 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:50.489+0000 7f08c4a15440 -1 Falling back to public interface 2024-09-15T03:04:50.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:04:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:52.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:52.952 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:04:52.951+0000 7f08c4a15440 -1 osd.2 94 log_to_monitors true 2024-09-15T03:04:53.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:53.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:53.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:04:53.046 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:53.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:53.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:53.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:04:54.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:04:54.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:04:54.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:04:54.448 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:04:54.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:04:54.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 up in weight 1 up_from 99 up_thru 85 down_at 96 last_clean_interval [72,94) [v2:127.0.0.1:6818/237849,v1:127.0.0.1:6819/237849] [v2:127.0.0.1:6820/237849,v1:127.0.0.1:6821/237849] exists,up 5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:04:54.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:04:54.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:04:54.845 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:04:54.845 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:04:54.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:04:54.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:04:54.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:04:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:04:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:04:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:04:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:04:54.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:04:54.934 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:04:55.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:04:55.336 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:04:55.336 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:04:55.337 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:04:55.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:04:55.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:55.337 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:55.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803837 2024-09-15T03:04:55.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803837 2024-09-15T03:04:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837' 2024-09-15T03:04:55.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:55.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:04:55.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=382252089352 2024-09-15T03:04:55.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 382252089352 2024-09-15T03:04:55.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837 1-382252089352' 2024-09-15T03:04:55.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:55.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:04:55.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=425201762307 2024-09-15T03:04:55.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 425201762307 2024-09-15T03:04:55.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837 1-382252089352 2-425201762307' 2024-09-15T03:04:55.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:04:55.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:04:55.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084329 2024-09-15T03:04:55.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084329 2024-09-15T03:04:55.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837 1-382252089352 2-425201762307 3-120259084329' 2024-09-15T03:04:55.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:55.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803837 2024-09-15T03:04:55.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:55.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:04:55.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803837 2024-09-15T03:04:55.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:55.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803837 2024-09-15T03:04:55.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803837' 2024-09-15T03:04:55.768 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803837 2024-09-15T03:04:55.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:56.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803835 -lt 25769803837 2024-09-15T03:04:56.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:04:57.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:04:57.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:04:57.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803837 -lt 25769803837 2024-09-15T03:04:57.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:57.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-382252089352 2024-09-15T03:04:57.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:57.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:04:57.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-382252089352 2024-09-15T03:04:57.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:57.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=382252089352 2024-09-15T03:04:57.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 382252089352' 2024-09-15T03:04:57.473 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 382252089352 2024-09-15T03:04:57.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:04:57.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 382252089352 -lt 382252089352 2024-09-15T03:04:57.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:57.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-425201762307 2024-09-15T03:04:57.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:57.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:04:57.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-425201762307 2024-09-15T03:04:57.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:57.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=425201762307 2024-09-15T03:04:57.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 425201762307' 2024-09-15T03:04:57.827 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 425201762307 2024-09-15T03:04:57.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:04:58.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 425201762307 -lt 425201762307 2024-09-15T03:04:58.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:04:58.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084329 2024-09-15T03:04:58.355 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:04:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:04:58.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084329 2024-09-15T03:04:58.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:04:58.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084329 2024-09-15T03:04:58.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084329' 2024-09-15T03:04:58.357 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084329 2024-09-15T03:04:58.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:04:58.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084329 -lt 120259084329 2024-09-15T03:04:58.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:04:58.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:58.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:04:59.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:04:59.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:04:59.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:04:59.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:04:59.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:04:59.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:04:59.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:04:59.233 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:04:59.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:04:59.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:04:59.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:04:59.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:05:00.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:05:00.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:05:00.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:05:00.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:256 bytes copied, 3.2911e-05 s, 7.8 MB/s 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-15T03:05:00.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:05:00.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:05:00.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:05:00.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:05:00.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:05:00.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:05:00.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-15T03:05:00.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:05:00.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T03:05:00.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-87967-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:05:02.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:02.615 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:05:02.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:05:02.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:05:02.617 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.2 2024-09-15T03:05:02.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:05:02.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T03:05:02.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:05:02.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:05:02.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:05:02.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:05:02.641 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:02.640+0000 7f5314b31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:02.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:02.642+0000 7f5314b31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:02.644 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:02.644+0000 7f5314b31440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:03.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:05:03.162 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:03.161+0000 7f5314b31440 -1 Falling back to public interface 2024-09-15T03:05:03.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:04.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:04.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:04.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:05:04.420 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:05:04.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:04.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:05:04.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:05.772 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:05.771+0000 7f5314b31440 -1 osd.2 100 log_to_monitors true 2024-09-15T03:05:05.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:05.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:05.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:05:05.777 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:05:05.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:05.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:05:06.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:07.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:07.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:07.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:05:07.253 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:05:07.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:07.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 up in weight 1 up_from 104 up_thru 85 down_at 101 last_clean_interval [99,100) [v2:127.0.0.1:6818/239206,v1:127.0.0.1:6819/239206] [v2:127.0.0.1:6820/239206,v1:127.0.0.1:6821/239206] exists,up 5bae0593-be68-41cf-a3d6-26caf057c493 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:05:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:05:07.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:05:07.756 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:05:07.756 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:05:07.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:05:07.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:05:07.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:05:07.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:05:07.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:05:07.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:05:07.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:05:07.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:05:07.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:05:08.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:05:08.192 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:05:08.193 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:05:08.193 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:05:08.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:05:08.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:05:08.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:05:08.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803842 2024-09-15T03:05:08.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803842 2024-09-15T03:05:08.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842' 2024-09-15T03:05:08.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:05:08.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:05:08.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=382252089356 2024-09-15T03:05:08.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 382252089356 2024-09-15T03:05:08.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842 1-382252089356' 2024-09-15T03:05:08.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:05:08.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:05:08.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=446676598787 2024-09-15T03:05:08.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 446676598787 2024-09-15T03:05:08.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842 1-382252089356 2-446676598787' 2024-09-15T03:05:08.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:05:08.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:05:08.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084334 2024-09-15T03:05:08.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084334 2024-09-15T03:05:08.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842 1-382252089356 2-446676598787 3-120259084334' 2024-09-15T03:05:08.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:05:08.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803842 2024-09-15T03:05:08.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:05:08.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:05:08.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803842 2024-09-15T03:05:08.631 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:05:08.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803842 2024-09-15T03:05:08.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-15T03:05:08.633 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803842 2024-09-15T03:05:08.633 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:05:08.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803839 -lt 25769803842 2024-09-15T03:05:08.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:05:09.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:05:09.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:05:10.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-15T03:05:10.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:05:10.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-382252089356 2024-09-15T03:05:10.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:05:10.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:05:10.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-382252089356 2024-09-15T03:05:10.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:05:10.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=382252089356 2024-09-15T03:05:10.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 382252089356' 2024-09-15T03:05:10.344 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 382252089356 2024-09-15T03:05:10.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:05:10.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 382252089356 -lt 382252089356 2024-09-15T03:05:10.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:05:10.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-446676598787 2024-09-15T03:05:10.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:05:10.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:05:10.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-446676598787 2024-09-15T03:05:10.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:05:10.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=446676598787 2024-09-15T03:05:10.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 446676598787' 2024-09-15T03:05:10.698 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 446676598787 2024-09-15T03:05:10.698 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:05:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 446676598787 -lt 446676598787 2024-09-15T03:05:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:05:11.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084334 2024-09-15T03:05:11.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:05:11.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:05:11.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084334 2024-09-15T03:05:11.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:05:11.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084334 2024-09-15T03:05:11.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084334' 2024-09-15T03:05:11.064 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 120259084334 2024-09-15T03:05:11.065 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:05:11.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084334 -lt 120259084334 2024-09-15T03:05:11.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:05:11.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:05:11.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:05:11.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:05:12.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:05:12.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:05:12.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:05:12.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:05:12.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:05:12.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:05:12.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:05:12.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-15T03:05:12.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-15T03:05:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-15T03:05:13.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-87967-1-256 fail 2024-09-15T03:05:13.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:05:13.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:05:13.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-87967-1-256 2024-09-15T03:05:13.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T03:05:13.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T03:05:13.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-87967-1-256 td/test-erasure-eio/COPY 2024-09-15T03:05:13.605 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-size-87967-1-256: (5) Input/output error 2024-09-15T03:05:13.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T03:05:13.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-15T03:05:13.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-15T03:05:13.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:05:13.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T03:05:14.170 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T03:05:14.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T03:05:14.785 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T03:05:14.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T03:05:14.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:05:14.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:05:14.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:05:14.797 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:05:14.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:05:14.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:05:14.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:05:14.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:05:14.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:05:14.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:05:14.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:05:14.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:05:14.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:05:14.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:05:14.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:05:14.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:14.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:05:14.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:05:14.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:14.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:05:14.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:05:14.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:05:14.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:05:14.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:14.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:05:14.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:05:14.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:05:14.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T03:05:14.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T03:05:14.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T03:05:14.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T03:05:14.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:05:14.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:05:14.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:05:14.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:05:14.945 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:05:14.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:05:14.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:05:14.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:05:14.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:14.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:05:14.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:05:14.948 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:05:14.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:05:14.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:05:14.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:05:14.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:05:14.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:14.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:14.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:05:14.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:05:14.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:05:14.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T03:05:14.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:05:14.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:14.953 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:14.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:05:14.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:05:14.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T03:05:14.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T03:05:15.313 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:05:15.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:15.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:15.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:05:15.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:05:15.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:05:15.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:05:15.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:05:15.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:05:15.567 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:05:15.567 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:05:15.567 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:05:15.567 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:05:15.567 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:05:15.567 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:15.568 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:15.568 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:05:15.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:05:15.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:05:15.646 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:05:15.647 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:05:15.647 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:15.647 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:15.647 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:05:15.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:05:15.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T03:05:15.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T03:05:15.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T03:05:15.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:05:15.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:05:15.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:05:15.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T03:05:15.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:05:16.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:05:16.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:16.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:16.560 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:16.560 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:16.560 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:16.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:16.560 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:05:16.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:05:16.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T03:05:16.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:05:16.842 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T03:05:16.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:05:17.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T03:05:17.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T03:05:17.921 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:05:17.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T03:05:17.930 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:05:15.340+0000 7fb40b167900 0 load: jerasure load: lrc 2024-09-15T03:05:17.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2024-09-15T03:05:17.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:255: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2024-09-15T03:05:17.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:256: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2024-09-15T03:05:17.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T03:05:17.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T03:05:17.931 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T03:05:17.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T03:05:17.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:05:17.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T03:05:17.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:05:17.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:05:17.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:05:17.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:05:17.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:05:17.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:05:17.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:05:17.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:05:17.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T03:05:17.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:05:17.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fb4fcebd-df1b-4129-bada-a7aec11d6afc 2024-09-15T03:05:17.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fb4fcebd-df1b-4129-bada-a7aec11d6afc' 2024-09-15T03:05:17.939 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 fb4fcebd-df1b-4129-bada-a7aec11d6afc 2024-09-15T03:05:17.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:05:17.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtTuZmtBKwOBAAU/ZOiQsiwC2b2UyH+Z9jHQ== 2024-09-15T03:05:17.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtTuZmtBKwOBAAU/ZOiQsiwC2b2UyH+Z9jHQ=="}' 2024-09-15T03:05:17.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fb4fcebd-df1b-4129-bada-a7aec11d6afc -i td/test-erasure-eio/0/new.json 2024-09-15T03:05:18.221 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:05:18.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T03:05:18.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtTuZmtBKwOBAAU/ZOiQsiwC2b2UyH+Z9jHQ== --osd-uuid fb4fcebd-df1b-4129-bada-a7aec11d6afc 2024-09-15T03:05:18.256 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:18.254+0000 7f8f0c81d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:18.267 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:18.265+0000 7f8f0c81d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:18.268 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:18.266+0000 7f8f0c81d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:18.269 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:18.267+0000 7f8f0c81d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:05:18.269 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:18.267+0000 7f8f0c81d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:05:18.269 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:18.267+0000 7f8f0c81d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:05:18.307 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:18.305+0000 7f8f0c81d440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T03:05:24.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T03:05:24.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:05:24.103 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T03:05:24.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:05:24.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:05:24.588 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T03:05:24.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:05:24.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:05:24.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:05:24.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:05:24.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:05:24.611 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:24.610+0000 7f4ac8c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:24.613 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:24.612+0000 7f4ac8c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:24.614 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:24.613+0000 7f4ac8c09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:24.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:05:24.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:05:24.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:05:24.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:05:24.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:05:24.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:24.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:05:24.937 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:05:24.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:24.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:25.126 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:25.125+0000 7f4ac8c09440 -1 Falling back to public interface 2024-09-15T03:05:25.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:26.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:26.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:26.302 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:05:26.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:05:26.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:26.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:26.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:27.623 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:27.622+0000 7f4ac8c09440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:05:27.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:27.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:27.651 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:05:27.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:05:27.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:27.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:28.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:29.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:29.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:29.008 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:05:29.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:05:29.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:29.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:29.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:30.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:30.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:30.469 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:05:30.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:05:30.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:30.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:30.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:31.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:31.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:31.874 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:05:31.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:05:31.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:31.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:32.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:33.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:33.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:33.314 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:05:33.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:05:33.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:33.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:33.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:34.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:34.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:34.713 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:05:34.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:05:34.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:34.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:35.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:36.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:05:36.097 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:05:36.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:36.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:36.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:37.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:37.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:37.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:05:37.469 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:05:37.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:37.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:37.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:38.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:38.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:38.874 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:05:38.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:05:38.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:38.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:39.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:40.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:40.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:40.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:05:40.437 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:05:40.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:40.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:41.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:42.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:42.114 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:05:42.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:42.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:05:42.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:42.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:42.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:43.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:43.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:43.639 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:05:43.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:05:43.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:43.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:44.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:45.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:45.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:45.504 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:05:45.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:05:45.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:45.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:46.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:47.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:47.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:47.319 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:05:47.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:05:47.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:47.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:48.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:49.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:49.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:49.068 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:05:49.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:05:49.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:49.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:49.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:50.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:50.827 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:05:50.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:50.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:05:50.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:50.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:05:51.255 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/241398,v1:127.0.0.1:6803/241398] [v2:127.0.0.1:6804/241398,v1:127.0.0.1:6805/241398] exists,up fb4fcebd-df1b-4129-bada-a7aec11d6afc 2024-09-15T03:05:51.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:05:51.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:05:51.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:05:51.261 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:05:51.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:05:51.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a27ffcf9-952b-497c-a388-242ee821b75f 2024-09-15T03:05:51.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a27ffcf9-952b-497c-a388-242ee821b75f' 2024-09-15T03:05:51.263 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 a27ffcf9-952b-497c-a388-242ee821b75f 2024-09-15T03:05:51.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:05:51.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPT+ZmgXJBEBAAICeGslHAukjJwzvpEUFkmw== 2024-09-15T03:05:51.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPT+ZmgXJBEBAAICeGslHAukjJwzvpEUFkmw=="}' 2024-09-15T03:05:51.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a27ffcf9-952b-497c-a388-242ee821b75f -i td/test-erasure-eio/1/new.json 2024-09-15T03:05:51.740 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:05:51.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T03:05:51.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPT+ZmgXJBEBAAICeGslHAukjJwzvpEUFkmw== --osd-uuid a27ffcf9-952b-497c-a388-242ee821b75f 2024-09-15T03:05:51.775 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:51.774+0000 7fc52d521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:51.777 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:51.776+0000 7fc52d521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:51.778 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:51.777+0000 7fc52d521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:51.778 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:51.777+0000 7fc52d521440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:05:51.779 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:51.777+0000 7fc52d521440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:05:51.779 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:51.777+0000 7fc52d521440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:05:51.855 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:51.854+0000 7fc52d521440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T03:05:57.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T03:05:57.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:05:57.743 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T03:05:57.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:05:57.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:05:58.236 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T03:05:58.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:05:58.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:05:58.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:05:58.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:05:58.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:05:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:58.258+0000 7f18dd523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:58.284 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:58.283+0000 7f18dd523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:58.285 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:58.284+0000 7f18dd523440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:05:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:05:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:05:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:05:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:05:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:05:58.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:58.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:05:58.631 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:05:58.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:58.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:05:58.814 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:05:58.813+0000 7f18dd523440 -1 Falling back to public interface 2024-09-15T03:05:58.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:05:59.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:05:59.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:05:59.985 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:05:59.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:05:59.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:05:59.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:00.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:01.337 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:01.336+0000 7f18dd523440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:06:01.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:01.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:01.349 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:06:01.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:06:01.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:01.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:01.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:02.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:02.848 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:06:02.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:02.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:06:02.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:02.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:03.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:04.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:04.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:04.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:06:04.279 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:06:04.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:04.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:04.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:05.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:05.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:05.687 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:06:05.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:06:05.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:05.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:06.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:07.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:07.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:07.186 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:06:07.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:06:07.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:07.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:07.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:08.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:08.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:08.609 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:06:08.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:06:08.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:08.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:09.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:10.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:10.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:10.042 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:06:10.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:06:10.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:10.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:10.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:11.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:11.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:11.464 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:06:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:06:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:11.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:12.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:12.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:12.844 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:06:12.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:06:12.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:12.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:13.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:14.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:14.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:14.219 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:06:14.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:06:14.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:14.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:15.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:16.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:16.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:16.522 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:06:16.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:06:16.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:16.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:17.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:18.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:18.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:18.412 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:06:18.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:06:18.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:18.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:19.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:20.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:20.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:20.198 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:06:20.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:06:20.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:20.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:21.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:22.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:22.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:22.194 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:06:22.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:06:22.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:22.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:22.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:23.679 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:06:23.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:06:23.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:23.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:24.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:25.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:25.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:25.456 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:06:25.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:06:25.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:25.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:06:25.810 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/243152,v1:127.0.0.1:6811/243152] [v2:127.0.0.1:6812/243152,v1:127.0.0.1:6813/243152] exists,up a27ffcf9-952b-497c-a388-242ee821b75f 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:06:25.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:06:25.815 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:06:25.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:06:25.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:06:25.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:06:25.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:06:25.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:06:25.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01dee9a7-89a9-4a7e-87b1-e362476c2ee0 2024-09-15T03:06:25.817 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 01dee9a7-89a9-4a7e-87b1-e362476c2ee0 2024-09-15T03:06:25.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 01dee9a7-89a9-4a7e-87b1-e362476c2ee0' 2024-09-15T03:06:25.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:06:25.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxT+Zm3FRPMRAAxKCRBLukkb4YyTwpX0o/qQ== 2024-09-15T03:06:25.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxT+Zm3FRPMRAAxKCRBLukkb4YyTwpX0o/qQ=="}' 2024-09-15T03:06:25.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01dee9a7-89a9-4a7e-87b1-e362476c2ee0 -i td/test-erasure-eio/2/new.json 2024-09-15T03:06:26.321 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:06:26.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T03:06:26.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxT+Zm3FRPMRAAxKCRBLukkb4YyTwpX0o/qQ== --osd-uuid 01dee9a7-89a9-4a7e-87b1-e362476c2ee0 2024-09-15T03:06:26.357 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:26.356+0000 7f153032f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:26.380 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:26.379+0000 7f153032f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:26.381 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:26.380+0000 7f153032f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:26.381 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:26.380+0000 7f153032f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:06:26.381 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:26.380+0000 7f153032f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:06:26.381 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:26.380+0000 7f153032f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:06:26.463 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:26.462+0000 7f153032f440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T03:06:32.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T03:06:32.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:06:32.238 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T03:06:32.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:06:32.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:06:32.721 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:06:32.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:06:32.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:06:32.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:06:32.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:06:32.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:06:32.744 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:32.743+0000 7f7551f5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:32.746 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:32.745+0000 7f7551f5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:32.747 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:32.746+0000 7f7551f5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:06:33.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:06:33.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:06:33.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:06:33.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:06:33.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:06:33.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:33.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:06:33.068 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:06:33.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:33.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:33.263 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:33.262+0000 7f7551f5e440 -1 Falling back to public interface 2024-09-15T03:06:33.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:34.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:34.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:34.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:06:34.417 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:06:34.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:34.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:34.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:35.737 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:06:35.736+0000 7f7551f5e440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:06:35.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:35.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:35.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:06:35.780 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:06:35.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:35.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:36.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:37.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:37.296 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:06:37.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:37.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:06:37.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:37.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:37.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:38.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:38.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:38.680 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:06:38.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:06:38.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:38.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:39.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:40.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:40.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:40.152 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:06:40.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:06:40.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:40.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:40.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:41.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:41.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:41.584 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:06:41.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:06:41.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:41.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:41.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:42.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:42.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:42.945 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:06:42.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:06:42.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:42.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:43.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:44.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:44.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:44.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:06:44.369 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:06:44.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:44.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:44.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:45.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:45.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:45.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:06:45.757 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:06:45.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:45.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:46.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:47.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:47.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:47.170 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:06:47.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:06:47.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:47.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:47.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:48.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:48.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:48.642 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:06:48.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:06:48.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:48.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:49.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:50.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:50.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:50.352 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:06:50.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:06:50.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:50.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:51.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:52.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:52.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:52.727 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:06:52.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:06:52.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:52.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:53.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:54.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:54.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:54.660 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:06:54.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:06:54.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:54.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:55.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:56.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:56.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:56.471 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:06:56.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:06:56.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:56.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:57.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:06:58.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:06:58.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:06:58.545 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:06:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:06:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:06:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:06:59.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:00.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:00.195 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:07:00.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:00.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:07:00.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:00.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:07:00.722 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/244907,v1:127.0.0.1:6819/244907] [v2:127.0.0.1:6820/244907,v1:127.0.0.1:6821/244907] exists,up 01dee9a7-89a9-4a7e-87b1-e362476c2ee0 2024-09-15T03:07:00.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:07:00.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:07:00.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:07:00.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:07:00.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T03:07:00.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:07:00.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:07:00.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:07:00.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=feb8ee8f-36f0-4f74-b7d0-1150671adcf7 2024-09-15T03:07:00.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 feb8ee8f-36f0-4f74-b7d0-1150671adcf7' 2024-09-15T03:07:00.730 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 feb8ee8f-36f0-4f74-b7d0-1150671adcf7 2024-09-15T03:07:00.730 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:07:00.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUT+ZmCgQWLBAAad6F759i8N6jTHPpVpiZcQ== 2024-09-15T03:07:00.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUT+ZmCgQWLBAAad6F759i8N6jTHPpVpiZcQ=="}' 2024-09-15T03:07:00.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new feb8ee8f-36f0-4f74-b7d0-1150671adcf7 -i td/test-erasure-eio/3/new.json 2024-09-15T03:07:01.637 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:07:01.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T03:07:01.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUT+ZmCgQWLBAAad6F759i8N6jTHPpVpiZcQ== --osd-uuid feb8ee8f-36f0-4f74-b7d0-1150671adcf7 2024-09-15T03:07:01.673 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:01.672+0000 7fbcc6833440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:01.675 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:01.674+0000 7fbcc6833440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:01.677 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:01.675+0000 7fbcc6833440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:01.677 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:01.675+0000 7fbcc6833440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:07:01.677 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:01.675+0000 7fbcc6833440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:07:01.677 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:01.675+0000 7fbcc6833440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:07:01.727 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:01.727+0000 7fbcc6833440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T03:07:07.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T03:07:07.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:07:07.933 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T03:07:07.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T03:07:07.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:07:08.436 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T03:07:08.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T03:07:08.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:07:08.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:07:08.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:07:08.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:07:08.459 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:08.457+0000 7f2449c5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:08.459+0000 7f2449c5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:08.462 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:08.460+0000 7f2449c5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T03:07:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:07:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T03:07:08.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:07:08.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:07:08.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:08.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:07:08.787 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:07:08.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:08.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:08.982 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:08.980+0000 7f2449c5a440 -1 Falling back to public interface 2024-09-15T03:07:09.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:10.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:10.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:10.156 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:07:10.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:07:10.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:10.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:10.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:11.479 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:11.477+0000 7f2449c5a440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:07:11.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:11.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:11.513 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:07:11.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:07:11.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:11.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:11.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:12.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:12.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:12.993 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:07:12.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:07:12.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:12.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:13.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:14.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:14.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:14.399 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:07:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:07:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:14.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:14.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:15.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:15.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:15.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:07:15.839 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:07:15.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:15.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:16.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:17.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:17.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:17.381 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:07:17.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:07:17.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:17.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:17.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:18.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:18.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:18.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:07:18.742 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:07:18.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:18.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:19.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:20.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:20.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:20.145 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:07:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:07:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:20.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:20.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:21.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:21.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:21.673 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:07:21.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:07:21.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:21.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:22.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:23.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:23.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:23.061 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:07:23.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:07:23.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:23.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:23.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:24.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:24.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:24.520 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:07:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:07:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:24.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:24.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:25.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:25.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:25.878 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:07:25.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:07:25.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:25.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:26.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:27.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:27.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:27.325 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:07:27.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:07:27.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:27.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:28.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:29.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:29.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:29.078 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:07:29.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:07:29.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:29.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:29.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:30.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:30.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:30.822 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:07:30.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:07:30.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:30.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:31.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:32.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:32.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:32.533 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:07:32.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:07:32.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:32.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:33.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:34.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:34.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:34.780 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:07:34.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:07:34.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:34.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:35.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:36.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:36.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:36.302 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T03:07:36.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:07:36.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:36.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:07:36.794 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/246661,v1:127.0.0.1:6827/246661] [v2:127.0.0.1:6828/246661,v1:127.0.0.1:6829/246661] exists,up feb8ee8f-36f0-4f74-b7d0-1150671adcf7 2024-09-15T03:07:36.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T03:07:36.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T03:07:36.870 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:07:36.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T03:07:36.881 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:05:25.129+0000 7f4ac8c09440 0 load: jerasure load: lrc 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:259: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T03:07:36.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T03:07:36.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T03:07:37.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T03:07:37.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T03:07:38.885 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T03:07:38.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:07:39.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T03:07:39.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:07:39.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:07:39.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:07:39.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:07:39.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:07:39.899 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:07:39.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:07:39.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:07:39.899 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:07:39.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:07:39.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:07:39.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:07:39.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:07:39.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:07:39.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:07:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:07:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:07:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:07:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:07:40.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:07:40.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:07:40.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:07:40.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803803 2024-09-15T03:07:40.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803803 2024-09-15T03:07:40.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803' 2024-09-15T03:07:40.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:07:40.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:07:40.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574871 2024-09-15T03:07:40.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574871 2024-09-15T03:07:40.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574871' 2024-09-15T03:07:40.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:07:40.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:07:40.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378641 2024-09-15T03:07:40.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378641 2024-09-15T03:07:40.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574871 2-81604378641' 2024-09-15T03:07:40.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:07:40.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:07:40.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051592 2024-09-15T03:07:40.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051592 2024-09-15T03:07:40.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574871 2-81604378641 3-124554051592' 2024-09-15T03:07:40.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:07:40.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803803 2024-09-15T03:07:40.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:07:40.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:07:40.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803803 2024-09-15T03:07:40.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:07:40.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803803 2024-09-15T03:07:40.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-09-15T03:07:40.827 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803803 2024-09-15T03:07:40.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:07:41.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803801 -lt 25769803803 2024-09-15T03:07:41.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:07:42.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:07:42.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:07:42.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803804 -lt 25769803803 2024-09-15T03:07:42.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:07:42.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574871 2024-09-15T03:07:42.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:07:42.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:07:42.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574871 2024-09-15T03:07:42.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:07:42.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574871 2024-09-15T03:07:42.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-09-15T03:07:42.526 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574871 2024-09-15T03:07:42.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:07:42.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574871 -lt 55834574871 2024-09-15T03:07:42.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:07:42.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378641 2024-09-15T03:07:42.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:07:42.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:07:42.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378641 2024-09-15T03:07:42.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:07:42.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378641 2024-09-15T03:07:42.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378641' 2024-09-15T03:07:42.878 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378641 2024-09-15T03:07:42.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:07:43.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378641 -lt 81604378641 2024-09-15T03:07:43.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:07:43.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051592 2024-09-15T03:07:43.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:07:43.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:07:43.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051592 2024-09-15T03:07:43.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:07:43.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051592 2024-09-15T03:07:43.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051592' 2024-09-15T03:07:43.258 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 124554051592 2024-09-15T03:07:43.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:07:43.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051593 -lt 124554051592 2024-09-15T03:07:43.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:07:43.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:07:43.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:07:44.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:07:44.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:07:44.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:07:44.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:07:44.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:07:44.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:07:44.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:07:44.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:07:44.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:07:44.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:07:44.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:07:44.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:07:44.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:261: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-15T03:07:44.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87967-0 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87967-0 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87967-0 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:07:44.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87967-0 td/test-erasure-eio/ORIGINAL 2024-09-15T03:07:45.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87967-0 td/test-erasure-eio 0 2024-09-15T03:07:45.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T03:07:45.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T03:07:45.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T03:07:45.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T03:07:45.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-87967-0 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-87967-0 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-0 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-0 2024-09-15T03:07:45.060 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:07:45.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:07:45.426 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:07:45.426 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:07:45.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:07:45.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T03:07:45.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T03:07:45.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T03:07:45.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T03:07:45.427 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:45.429 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:45.430 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:45.430 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T03:07:45.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T03:07:45.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:07:45.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T03:07:45.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T03:07:45.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T03:07:45.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T03:07:45.509 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T03:07:45.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:45.509 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:45.510 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:45.510 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T03:07:45.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T03:07:45.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-87967-0 0 2024-09-15T03:07:45.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87967-0 2024-09-15T03:07:45.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:07:45.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:07:45.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87967-0 2024-09-15T03:07:45.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:07:45.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:07:45.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87967-0 td/test-erasure-eio/COPY 2024-09-15T03:07:45.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:07:45.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:07:45.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-15T03:07:45.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2024-09-15T03:07:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=1 2024-09-15T03:07:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87967-0 td/test-erasure-eio 1 2024-09-15T03:07:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T03:07:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T03:07:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T03:07:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T03:07:45.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-87967-0 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-87967-0 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-0 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-0 2024-09-15T03:07:45.646 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:07:45.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:07:45.996 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:07:45.996 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:07:45.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:07:45.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T03:07:45.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T03:07:45.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T03:07:45.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T03:07:45.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:07:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T03:07:46.000 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:07:46.000 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:07:46.000 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:46.000 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:46.000 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:46.000 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T03:07:46.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T03:07:46.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:07:46.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T03:07:46.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-87967-0 1 2024-09-15T03:07:46.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87967-0 fail 2024-09-15T03:07:46.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:07:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:07:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87967-0 2024-09-15T03:07:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T03:07:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T03:07:46.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-87967-0 td/test-erasure-eio/COPY 2024-09-15T03:07:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-eio-87967-0: (5) Input/output error 2024-09-15T03:07:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T03:07:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-15T03:07:46.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:263: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-15T03:07:46.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:07:46.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T03:07:46.745 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T03:07:46.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T03:07:47.331 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T03:07:47.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:07:47.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:07:47.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:07:47.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:07:47.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:07:47.453 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:07:47.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:07:47.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:07:47.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:07:47.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:47.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:07:47.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:07:47.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:47.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:07:47.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:07:47.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:07:47.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:07:47.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:47.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:47.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:07:47.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:07:47.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:07:47.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T03:07:47.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T03:07:47.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:07:47.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:07:47.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:07:47.481 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:07:47.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:07:47.482 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:07:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:07:47.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:07:47.483 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:07:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:07:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:07:47.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:07:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:07:47.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:07:47.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:07:47.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:07:47.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:47.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:47.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:07:47.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:07:47.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:07:47.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T03:07:47.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:07:47.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:47.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:47.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T03:07:47.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:07:47.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:07:48.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:07:48.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:07:48.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:07:48.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:07:48.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:07:48.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:07:48.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T03:07:48.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:07:48.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T03:07:48.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T03:07:48.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T03:07:48.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:07:48.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:07:48.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:07:48.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T03:07:48.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:07:49.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:07:49.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:49.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:49.116 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:49.116 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:49.116 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:49.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:07:49.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:07:49.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:07:49.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T03:07:49.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:07:49.407 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T03:07:49.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T03:07:50.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T03:07:50.486 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:07:50.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T03:07:50.495 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:07:47.881+0000 7f42bca0a900 0 load: jerasure load: lrc 2024-09-15T03:07:50.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2024-09-15T03:07:50.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:267: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2024-09-15T03:07:50.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:268: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2024-09-15T03:07:50.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T03:07:50.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T03:07:50.496 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T03:07:50.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T03:07:50.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T03:07:50.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:07:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T03:07:50.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:07:50.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ed484c60-3ccf-427c-ae3f-285fa9202fc7 2024-09-15T03:07:50.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ed484c60-3ccf-427c-ae3f-285fa9202fc7' 2024-09-15T03:07:50.504 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 ed484c60-3ccf-427c-ae3f-285fa9202fc7 2024-09-15T03:07:50.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:07:50.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGT+ZmimmvHhAAi0q25sfBN/7kIyBedgEqQg== 2024-09-15T03:07:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGT+ZmimmvHhAAi0q25sfBN/7kIyBedgEqQg=="}' 2024-09-15T03:07:50.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ed484c60-3ccf-427c-ae3f-285fa9202fc7 -i td/test-erasure-eio/0/new.json 2024-09-15T03:07:50.785 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:07:50.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T03:07:50.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGT+ZmimmvHhAAi0q25sfBN/7kIyBedgEqQg== --osd-uuid ed484c60-3ccf-427c-ae3f-285fa9202fc7 2024-09-15T03:07:50.820 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:50.819+0000 7fde4888d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:50.823 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:50.822+0000 7fde4888d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:50.824 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:50.823+0000 7fde4888d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:50.825 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:50.824+0000 7fde4888d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:07:50.825 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:50.824+0000 7fde4888d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:07:50.825 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:50.824+0000 7fde4888d440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:07:50.855 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:50.854+0000 7fde4888d440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T03:07:56.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T03:07:56.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:07:56.921 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T03:07:56.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:07:56.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:07:57.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:07:57.401 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T03:07:57.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:07:57.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:07:57.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:07:57.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:07:57.423 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:57.422+0000 7fbb5da08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:57.425 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:57.424+0000 7fbb5da08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:57.426 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:57.425+0000 7fbb5da08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:07:57.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:07:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:07:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:07:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:07:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:07:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:07:57.835 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:07:57.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:57.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:57.947 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:07:57.946+0000 7fbb5da08440 -1 Falling back to public interface 2024-09-15T03:07:58.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:07:59.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:07:59.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:07:59.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:07:59.184 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:07:59.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:07:59.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:07:59.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:00.424 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:00.423+0000 7fbb5da08440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:08:00.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:00.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:00.540 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:08:00.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:08:00.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:00.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:00.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:01.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:01.978 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:08:01.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:01.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:08:01.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:01.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:02.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:03.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:03.439 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:08:03.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:08:03.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:03.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:03.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:04.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:04.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:04.800 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:08:04.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:08:04.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:04.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:05.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:06.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:06.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:06.173 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:08:06.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:08:06.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:06.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:06.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:07.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:07.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:07.573 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:08:07.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:08:07.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:07.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:08.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:09.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:09.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:09.112 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:08:09.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:08:09.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:09.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:09.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:10.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:10.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:10.490 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:08:10.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:08:10.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:10.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:10.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:11.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:11.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:11.897 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:08:11.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:08:11.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:11.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:12.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:13.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:13.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:13.617 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:08:13.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:08:13.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:13.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:14.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:15.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:15.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:15.428 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:08:15.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:08:15.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:15.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:16.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:17.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:17.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:17.164 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:08:17.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:08:17.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:17.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:17.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:18.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:18.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:18.908 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:08:18.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:08:18.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:18.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:20.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:20.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:20.726 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:08:20.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:08:20.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:20.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:21.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:22.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:22.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:22.277 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:08:22.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:08:22.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:22.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:22.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:23.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:23.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:23.825 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:08:23.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:08:23.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:23.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:08:24.184 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/249375,v1:127.0.0.1:6803/249375] [v2:127.0.0.1:6804/249375,v1:127.0.0.1:6805/249375] exists,up ed484c60-3ccf-427c-ae3f-285fa9202fc7 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:08:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:08:24.189 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:08:24.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:08:24.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:08:24.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:08:24.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:08:24.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1029f606-1d18-4058-a3ec-3f4e02dde96a 2024-09-15T03:08:24.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1029f606-1d18-4058-a3ec-3f4e02dde96a' 2024-09-15T03:08:24.191 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 1029f606-1d18-4058-a3ec-3f4e02dde96a 2024-09-15T03:08:24.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:08:24.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoT+Zmo8sHDBAAXu30TObLhWPgT5U83jvXlQ== 2024-09-15T03:08:24.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoT+Zmo8sHDBAAXu30TObLhWPgT5U83jvXlQ=="}' 2024-09-15T03:08:24.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1029f606-1d18-4058-a3ec-3f4e02dde96a -i td/test-erasure-eio/1/new.json 2024-09-15T03:08:24.672 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:08:24.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T03:08:24.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoT+Zmo8sHDBAAXu30TObLhWPgT5U83jvXlQ== --osd-uuid 1029f606-1d18-4058-a3ec-3f4e02dde96a 2024-09-15T03:08:24.708 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:24.707+0000 7f50af92b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:24.709 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:24.709+0000 7f50af92b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:24.710 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:24.710+0000 7f50af92b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:24.711 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:24.710+0000 7f50af92b440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:08:24.711 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:24.710+0000 7f50af92b440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:08:24.711 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:24.710+0000 7f50af92b440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:08:24.787 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:24.786+0000 7f50af92b440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T03:08:30.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T03:08:30.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:08:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:08:30.668 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T03:08:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:08:31.146 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T03:08:31.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:08:31.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:08:31.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:08:31.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:08:31.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:08:31.169 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:31.168+0000 7f1198c6f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:31.171 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:31.170+0000 7f1198c6f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:31.173 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:31.172+0000 7f1198c6f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:31.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:08:31.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:08:31.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:08:31.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:08:31.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:08:31.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:31.500 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:08:31.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:08:31.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:31.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:31.687 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:31.686+0000 7f1198c6f440 -1 Falling back to public interface 2024-09-15T03:08:31.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:32.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:32.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:32.866 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:08:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:08:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:32.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:33.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:34.163 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:34.161+0000 7f1198c6f440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:08:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:08:34.227 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:08:34.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:34.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:34.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:35.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:35.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:35.706 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:08:35.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:08:35.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:35.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:36.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:37.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:37.146 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:08:37.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:37.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:08:37.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:37.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:37.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:38.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:38.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:38.652 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:08:38.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:08:38.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:38.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:39.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:40.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:40.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:40.125 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:08:40.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:08:40.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:40.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:40.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:41.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:41.525 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:08:41.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:41.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:08:41.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:41.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:42.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:43.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:43.019 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:08:43.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:43.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:08:43.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:43.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:43.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:44.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:44.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:44.555 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:08:44.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:08:44.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:44.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:44.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:45.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:45.930 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:08:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:08:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:45.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:46.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:47.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:47.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:47.476 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:08:47.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:08:47.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:47.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:48.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:49.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:49.565 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:08:49.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:49.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:08:49.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:49.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:50.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:51.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:51.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:51.024 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:08:51.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:08:51.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:51.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:52.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:53.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:53.046 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:08:53.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:53.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:08:53.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:53.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:53.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:54.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:54.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:54.505 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:08:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:08:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:54.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:55.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:56.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:56.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:56.014 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:08:56.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:08:56.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:56.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:56.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:08:57.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:08:57.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:08:57.685 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:08:57.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:08:57.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:08:57.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:08:58.252 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/251129,v1:127.0.0.1:6811/251129] [v2:127.0.0.1:6812/251129,v1:127.0.0.1:6813/251129] exists,up 1029f606-1d18-4058-a3ec-3f4e02dde96a 2024-09-15T03:08:58.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:08:58.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:08:58.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:08:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:08:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:08:58.259 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:08:58.260 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 88545ff8-41c7-4724-b017-346a3a81505f 2024-09-15T03:08:58.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88545ff8-41c7-4724-b017-346a3a81505f 2024-09-15T03:08:58.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 88545ff8-41c7-4724-b017-346a3a81505f' 2024-09-15T03:08:58.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:08:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKT+ZmfUYfEBAAYL91FO2ks6UBr3FS8DxsRQ== 2024-09-15T03:08:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKT+ZmfUYfEBAAYL91FO2ks6UBr3FS8DxsRQ=="}' 2024-09-15T03:08:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88545ff8-41c7-4724-b017-346a3a81505f -i td/test-erasure-eio/2/new.json 2024-09-15T03:08:58.910 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:08:58.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T03:08:58.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKT+ZmfUYfEBAAYL91FO2ks6UBr3FS8DxsRQ== --osd-uuid 88545ff8-41c7-4724-b017-346a3a81505f 2024-09-15T03:08:58.946 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:58.944+0000 7ffaa051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:58.969 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:58.967+0000 7ffaa051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:58.970 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:58.968+0000 7ffaa051f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:08:58.970 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:58.968+0000 7ffaa051f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:08:58.971 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:58.968+0000 7ffaa051f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:08:58.971 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:58.968+0000 7ffaa051f440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:08:59.061 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:08:59.059+0000 7ffaa051f440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T03:09:04.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T03:09:04.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:09:04.891 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T03:09:04.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:09:04.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:09:05.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:09:05.403 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:09:05.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:09:05.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:09:05.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:09:05.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:09:05.426 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:05.424+0000 7ff3f668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:05.461 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:05.459+0000 7ff3f668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:05.462 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:05.460+0000 7ff3f668d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:05.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:09:05.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:09:05.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:09:05.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:09:05.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:09:05.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:05.756 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:09:05.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:09:05.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:05.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:05.983 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:05.981+0000 7ff3f668d440 -1 Falling back to public interface 2024-09-15T03:09:06.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:07.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:07.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:07.109 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:09:07.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:09:07.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:07.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:07.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:08.464 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:08.463+0000 7ff3f668d440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:09:08.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:08.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:08.470 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:09:08.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:09:08.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:08.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:08.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:09.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:09.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:09.917 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:09:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:09:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:09.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:10.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:11.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:11.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:11.363 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:09:11.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:09:11.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:11.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:11.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:12.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:12.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:12.836 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:09:12.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:09:12.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:12.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:13.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:14.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:14.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:14.333 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:09:14.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:09:14.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:14.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:14.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:15.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:15.694 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:09:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:09:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:15.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:16.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:17.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:17.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:17.056 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:09:17.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:09:17.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:17.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:17.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:18.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:18.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:18.522 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:09:18.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:09:18.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:18.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:18.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:19.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:19.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:19.938 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:09:19.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:09:19.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:19.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:20.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:21.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:21.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:21.323 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:09:21.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:09:21.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:21.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:21.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:22.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:22.885 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:09:22.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:22.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:09:22.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:22.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:24.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:25.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:25.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:25.091 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:09:25.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:09:25.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:25.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:25.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:26.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:26.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:26.606 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:09:26.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:09:26.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:26.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:27.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:28.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:28.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:28.097 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:09:28.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:09:28.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:28.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:28.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:29.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:29.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:29.812 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:09:29.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:09:29.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:29.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:30.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:31.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:31.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:31.565 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:09:31.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:09:31.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:31.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:09:32.002 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/252883,v1:127.0.0.1:6819/252883] [v2:127.0.0.1:6820/252883,v1:127.0.0.1:6821/252883] exists,up 88545ff8-41c7-4724-b017-346a3a81505f 2024-09-15T03:09:32.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:09:32.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:09:32.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:09:32.008 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:09:32.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:09:32.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T03:09:32.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:09:32.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a3954e1f-d0cf-402f-bc89-39efc39ae4a2 2024-09-15T03:09:32.010 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 a3954e1f-d0cf-402f-bc89-39efc39ae4a2 2024-09-15T03:09:32.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a3954e1f-d0cf-402f-bc89-39efc39ae4a2' 2024-09-15T03:09:32.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:09:32.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsT+Zm9UAzARAAphX01YOjekUiOP5F4VYOWA== 2024-09-15T03:09:32.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsT+Zm9UAzARAAphX01YOjekUiOP5F4VYOWA=="}' 2024-09-15T03:09:32.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a3954e1f-d0cf-402f-bc89-39efc39ae4a2 -i td/test-erasure-eio/3/new.json 2024-09-15T03:09:33.216 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:09:33.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T03:09:33.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsT+Zm9UAzARAAphX01YOjekUiOP5F4VYOWA== --osd-uuid a3954e1f-d0cf-402f-bc89-39efc39ae4a2 2024-09-15T03:09:33.252 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:33.251+0000 7f2385b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:33.253 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:33.253+0000 7f2385b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:33.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:33.253+0000 7f2385b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:33.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:33.254+0000 7f2385b1d440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:09:33.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:33.254+0000 7f2385b1d440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:09:33.255 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:33.254+0000 7f2385b1d440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:09:33.333 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:33.332+0000 7f2385b1d440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T03:09:39.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T03:09:39.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:09:39.451 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T03:09:39.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T03:09:39.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:09:39.935 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T03:09:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T03:09:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:09:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:09:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:09:39.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:09:39.958 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:39.957+0000 7f2e65821440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:39.960 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:39.959+0000 7f2e65821440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:39.961 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:39.960+0000 7f2e65821440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:09:40.283 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:09:40.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:40.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:40.483 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:40.482+0000 7f2e65821440 -1 Falling back to public interface 2024-09-15T03:09:40.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:41.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:41.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:41.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:09:41.637 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:09:41.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:41.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:41.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:42.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:42.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:42.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:09:42.994 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:09:42.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:42.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:43.007 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:09:43.006+0000 7f2e65821440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:09:43.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:44.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:44.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:44.468 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:09:44.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:09:44.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:44.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:44.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:45.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:45.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:45.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:09:45.940 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:09:45.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:45.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:46.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:47.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:47.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:47.321 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:09:47.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:09:47.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:47.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:47.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:48.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:48.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:48.776 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:09:48.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:09:48.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:48.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:49.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:50.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:50.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:50.186 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:09:50.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:09:50.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:50.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:50.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:51.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:51.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:51.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:09:51.572 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:09:51.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:51.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:52.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:53.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:53.042 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:09:53.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:53.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:09:53.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:53.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:53.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:54.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:54.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:54.487 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:09:54.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:09:54.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:54.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:55.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:56.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:56.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:56.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:09:56.021 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:09:56.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:56.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:56.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:57.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:57.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:57.653 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:09:57.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:09:57.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:57.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:09:58.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:09:59.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:09:59.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:09:59.260 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:09:59.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:09:59.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:09:59.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:10:00.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:01.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:01.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:01.007 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:10:01.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:10:01.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:01.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:10:01.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:02.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:02.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:02.911 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:10:02.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:10:02.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:02.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:10:03.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:04.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:04.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:04.760 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:10:04.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:10:04.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:04.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:10:05.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:06.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:06.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:06.205 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:10:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:10:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:10:07.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:08.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:08.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:08.139 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T03:10:08.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:10:08.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:08.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:10:08.580 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/254638,v1:127.0.0.1:6827/254638] [v2:127.0.0.1:6828/254638,v1:127.0.0.1:6829/254638] exists,up a3954e1f-d0cf-402f-bc89-39efc39ae4a2 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:08.582 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:08.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:08.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T03:10:08.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T03:10:08.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T03:10:08.656 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:10:08.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T03:10:08.666 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:07:57.950+0000 7fbb5da08440 0 load: jerasure load: lrc 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:270: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:271: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T03:10:08.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T03:10:08.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T03:10:09.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T03:10:09.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T03:10:10.748 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T03:10:10.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:10:11.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T03:10:11.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:10:11.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:10:11.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:10:11.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:10:11.763 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:10:11.763 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:10:11.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:10:11.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:10:11.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:10:11.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:10:11.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:10:11.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:10:11.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:10:11.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:10:11.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:10:12.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:10:12.212 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:10:12.212 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:10:12.212 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:10:12.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:10:12.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:10:12.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:10:12.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803805 2024-09-15T03:10:12.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803805 2024-09-15T03:10:12.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805' 2024-09-15T03:10:12.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:10:12.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:10:12.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574871 2024-09-15T03:10:12.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574871 2024-09-15T03:10:12.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805 1-55834574871' 2024-09-15T03:10:12.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:10:12.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:10:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378634 2024-09-15T03:10:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378634 2024-09-15T03:10:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805 1-55834574871 2-81604378634' 2024-09-15T03:10:12.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:10:12.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:10:12.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084296 2024-09-15T03:10:12.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084296 2024-09-15T03:10:12.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803805 1-55834574871 2-81604378634 3-120259084296' 2024-09-15T03:10:12.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:10:12.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803805 2024-09-15T03:10:12.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:10:12.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:10:12.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803805 2024-09-15T03:10:12.675 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:10:12.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803805 2024-09-15T03:10:12.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2024-09-15T03:10:12.676 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803805 2024-09-15T03:10:12.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:10:13.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803805 -lt 25769803805 2024-09-15T03:10:13.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:10:13.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574871 2024-09-15T03:10:13.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:10:13.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:10:13.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574871 2024-09-15T03:10:13.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:10:13.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574871 2024-09-15T03:10:13.033 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574871 2024-09-15T03:10:13.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-09-15T03:10:13.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:10:13.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574871 -lt 55834574871 2024-09-15T03:10:13.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:10:13.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378634 2024-09-15T03:10:13.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:10:13.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:10:13.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378634 2024-09-15T03:10:13.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:10:13.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378634 2024-09-15T03:10:13.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378634' 2024-09-15T03:10:13.441 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378634 2024-09-15T03:10:13.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:10:13.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378634 -lt 81604378634 2024-09-15T03:10:13.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:10:13.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084296 2024-09-15T03:10:13.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:10:13.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:10:13.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084296 2024-09-15T03:10:13.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:10:13.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084296 2024-09-15T03:10:13.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084296' 2024-09-15T03:10:13.834 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084296 2024-09-15T03:10:13.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:10:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084296 -lt 120259084296 2024-09-15T03:10:14.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:10:14.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:10:14.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:10:14.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:10:14.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:10:14.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:10:14.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:10:14.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:273: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:274: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T03:10:15.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87967-1 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87967-1 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87967-1 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:10:15.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:10:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:10:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:10:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:10:15.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87967-1 td/test-erasure-eio/ORIGINAL 2024-09-15T03:10:15.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87967-1 td/test-erasure-eio 1 2024-09-15T03:10:15.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T03:10:15.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-87967-1 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-87967-1 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-1 2024-09-15T03:10:15.626 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-1 2024-09-15T03:10:15.627 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:10:16.011 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T03:10:16.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T03:10:16.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T03:10:16.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T03:10:16.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T03:10:16.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T03:10:16.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T03:10:16.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:10:16.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:10:16.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T03:10:16.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:10:16.014 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:10:16.015 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:16.015 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:16.015 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:16.015 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T03:10:16.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T03:10:16.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:10:16.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:16.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T03:10:16.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T03:10:16.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-87967-1 1 2024-09-15T03:10:16.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87967-1 2024-09-15T03:10:16.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:10:16.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:10:16.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87967-1 2024-09-15T03:10:16.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:10:16.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:10:16.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87967-1 td/test-erasure-eio/COPY 2024-09-15T03:10:16.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:10:16.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:10:16.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-15T03:10:16.209 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87967-1 td/test-erasure-eio 2 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-87967-1 2024-09-15T03:10:16.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=2 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-87967-1 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-1 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-1 2024-09-15T03:10:16.213 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:10:16.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:10:16.558 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:10:16.558 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:10:16.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:10:16.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T03:10:16.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T03:10:16.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=2 2024-09-15T03:10:16.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T03:10:16.559 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/2/type 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:16.561 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:16.562 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.2.asok 2024-09-15T03:10:16.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-15T03:10:16.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:10:16.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T03:10:16.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T03:10:16.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.2 2024-09-15T03:10:16.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:10:16.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:10:16.642 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:16.643 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:16.643 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:16.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.2.asok 2024-09-15T03:10:16.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T03:10:16.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-87967-1 2 2024-09-15T03:10:16.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87967-1 fail 2024-09-15T03:10:16.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:10:16.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:10:16.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87967-1 2024-09-15T03:10:16.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T03:10:16.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T03:10:16.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-87967-1 td/test-erasure-eio/COPY 2024-09-15T03:10:16.781 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-eio-87967-1: (5) Input/output error 2024-09-15T03:10:16.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T03:10:16.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-15T03:10:16.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-15T03:10:16.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:10:16.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T03:10:17.404 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T03:10:17.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T03:10:18.029 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:10:18.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:10:18.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:10:18.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:10:18.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:10:18.149 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:10:18.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:10:18.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:10:18.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:10:18.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:18.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:10:18.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:10:18.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:18.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:10:18.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:10:18.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:10:18.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:10:18.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:18.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:18.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:10:18.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:10:18.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:10:18.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T03:10:18.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T03:10:18.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T03:10:18.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:10:18.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:10:18.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:10:18.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:10:18.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:10:18.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:10:18.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:10:18.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:10:18.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:18.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:10:18.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:10:18.180 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:10:18.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:10:18.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:10:18.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:10:18.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:10:18.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:18.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:18.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:10:18.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:10:18.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:10:18.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T03:10:18.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:10:18.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:18.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:18.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T03:10:18.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:10:18.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:10:18.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:10:18.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T03:10:18.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T03:10:18.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T03:10:18.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:10:18.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:10:18.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:10:18.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T03:10:18.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:10:18.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:10:18.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:10:18.817 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:10:18.818 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:18.818 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:18.818 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:18.818 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:10:18.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:10:18.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T03:10:18.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:10:18.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:10:18.892 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:18.893 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:10:18.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:10:18.894 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T03:10:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T03:10:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T03:10:18.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:10:18.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:10:18.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:10:18.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T03:10:18.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:10:19.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:10:19.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:10:19.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:10:19.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:10:19.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:19.803 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:19.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:10:19.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:10:19.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:10:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T03:10:19.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:10:20.077 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T03:10:20.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T03:10:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T03:10:21.156 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:10:21.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T03:10:21.164 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:10:18.583+0000 7f062b020900 0 load: jerasure load: lrc 2024-09-15T03:10:21.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2024-09-15T03:10:21.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2024-09-15T03:10:21.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_missing: setup_osds 4 2024-09-15T03:10:21.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T03:10:21.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:10:21.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:10:21.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:10:21.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c2046f9-fb7a-4720-82b3-464909a82d24 2024-09-15T03:10:21.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1c2046f9-fb7a-4720-82b3-464909a82d24' 2024-09-15T03:10:21.176 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 1c2046f9-fb7a-4720-82b3-464909a82d24 2024-09-15T03:10:21.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:10:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdUOZmAdj/ChAAyFDOSdQyofsEj8WoOPyizw== 2024-09-15T03:10:21.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdUOZmAdj/ChAAyFDOSdQyofsEj8WoOPyizw=="}' 2024-09-15T03:10:21.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c2046f9-fb7a-4720-82b3-464909a82d24 -i td/test-erasure-eio/0/new.json 2024-09-15T03:10:21.456 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:10:21.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T03:10:21.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdUOZmAdj/ChAAyFDOSdQyofsEj8WoOPyizw== --osd-uuid 1c2046f9-fb7a-4720-82b3-464909a82d24 2024-09-15T03:10:21.491 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:21.490+0000 7f9db7521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:21.501 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:21.501+0000 7f9db7521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:21.503 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:21.502+0000 7f9db7521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:21.503 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:21.502+0000 7f9db7521440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:10:21.503 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:21.502+0000 7f9db7521440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:10:21.503 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:21.502+0000 7f9db7521440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:10:21.542 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:21.541+0000 7f9db7521440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T03:10:27.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T03:10:27.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:10:27.575 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T03:10:27.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:10:27.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:10:28.052 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T03:10:28.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:10:28.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:10:28.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:10:28.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:10:28.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:10:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:28.074+0000 7f45c4e2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:28.077 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:28.076+0000 7f45c4e2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:28.078 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:28.077+0000 7f45c4e2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:10:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:10:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:10:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:10:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:10:28.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:28.402 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:10:28.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:10:28.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:28.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:28.592 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:28.591+0000 7f45c4e2d440 -1 Falling back to public interface 2024-09-15T03:10:28.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:29.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:29.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:29.757 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:10:29.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:10:29.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:29.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:30.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:31.080 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:31.079+0000 7f45c4e2d440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:10:31.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:31.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:31.111 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:10:31.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:10:31.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:31.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:31.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:10:32.496 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:10:32.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:32.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:32.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:33.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:33.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:33.937 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:10:33.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:10:33.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:33.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:34.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:35.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:35.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:35.316 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:10:35.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:10:35.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:35.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:35.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:36.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:36.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:36.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:10:36.748 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:10:36.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:36.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:37.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:38.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:38.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:38.108 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:10:38.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:10:38.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:38.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:38.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:39.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:39.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:39.500 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:10:39.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:10:39.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:39.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:39.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:40.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:40.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:40.904 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:10:40.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:10:40.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:40.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:41.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:42.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:42.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:42.276 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:10:42.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:10:42.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:42.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:42.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:43.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:43.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:43.684 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:10:43.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:10:43.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:43.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:44.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:45.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:45.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:45.502 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:10:45.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:10:45.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:45.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:46.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:47.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:47.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:47.495 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:10:47.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:10:47.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:47.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:47.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:48.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:48.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:48.909 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:10:48.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:10:48.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:48.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:49.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:50.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:50.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:50.642 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:10:50.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:10:50.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:50.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:51.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:52.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:52.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:52.672 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:10:52.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:10:52.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:52.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:53.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:10:54.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:10:54.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:10:54.221 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:10:54.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:10:54.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:10:54.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:10:54.702 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/257328,v1:127.0.0.1:6803/257328] [v2:127.0.0.1:6804/257328,v1:127.0.0.1:6805/257328] exists,up 1c2046f9-fb7a-4720-82b3-464909a82d24 2024-09-15T03:10:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:10:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:10:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:10:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:10:54.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T03:10:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:10:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:10:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:10:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:10:54.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:10:54.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:10:54.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:10:54.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:10:54.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:10:54.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:10:54.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:10:54.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:10:54.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:10:54.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:10:54.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:10:54.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:10:54.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:10:54.715 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:10:54.715 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:10:54.715 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:10:54.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:10:54.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:10:54.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:10:54.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:10:54.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:10:54.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:10:54.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:10:54.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:10:54.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:10:54.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:10:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:10:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:10:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:10:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:10:54.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:10:54.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1150adab-473a-42f5-a409-e1c88618fc18 2024-09-15T03:10:54.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1150adab-473a-42f5-a409-e1c88618fc18' 2024-09-15T03:10:54.719 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 1150adab-473a-42f5-a409-e1c88618fc18 2024-09-15T03:10:54.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:10:54.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+UOZmeLzZKhAAhaSjOUBkprFW4NFRoIHoCA== 2024-09-15T03:10:54.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+UOZmeLzZKhAAhaSjOUBkprFW4NFRoIHoCA=="}' 2024-09-15T03:10:54.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1150adab-473a-42f5-a409-e1c88618fc18 -i td/test-erasure-eio/1/new.json 2024-09-15T03:10:55.446 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:10:55.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T03:10:55.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+UOZmeLzZKhAAhaSjOUBkprFW4NFRoIHoCA== --osd-uuid 1150adab-473a-42f5-a409-e1c88618fc18 2024-09-15T03:10:55.481 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:55.479+0000 7fa0f948b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:55.504 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:55.502+0000 7fa0f948b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:55.505 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:55.503+0000 7fa0f948b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:10:55.506 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:55.504+0000 7fa0f948b440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:10:55.506 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:55.504+0000 7fa0f948b440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:10:55.506 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:55.504+0000 7fa0f948b440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:10:55.581 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:10:55.578+0000 7fa0f948b440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T03:11:01.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T03:11:01.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:11:01.420 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T03:11:01.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:11:01.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:11:01.901 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T03:11:01.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:11:01.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:11:01.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:11:01.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:11:01.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:11:01.924 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:01.923+0000 7fc2b0323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:01.925+0000 7fc2b0323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:01.927 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:01.926+0000 7fc2b0323440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:02.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:11:02.251 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:11:02.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:11:02.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:11:02.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:11:02.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:11:02.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:02.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:11:02.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:02.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:02.456 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:02.455+0000 7fc2b0323440 -1 Falling back to public interface 2024-09-15T03:11:02.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:03.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:03.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:03.610 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:11:03.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:11:03.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:03.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:03.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:04.939 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:04.938+0000 7fc2b0323440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:11:04.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:04.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:04.986 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:11:04.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:11:04.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:04.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:05.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:06.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:06.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:06.734 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:11:06.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:11:06.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:06.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:07.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:08.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:08.114 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:11:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:11:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:08.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:08.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:09.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:09.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:09.612 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:11:09.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:11:09.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:09.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:10.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:11.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:11.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:11.002 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:11:11.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:11:11.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:11.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:11.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:12.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:12.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:12.391 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:11:12.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:11:12.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:12.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:12.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:13.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:13.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:13.881 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:11:13.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:11:13.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:13.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:14.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:15.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:15.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:15.314 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:11:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:11:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:15.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:15.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:16.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:16.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:16.841 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:11:16.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:11:16.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:16.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:17.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:18.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:18.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:18.200 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:11:18.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:11:18.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:18.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:19.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:20.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:20.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:20.092 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:11:20.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:11:20.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:20.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:21.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:22.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:22.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:22.147 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:11:22.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:11:22.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:22.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:23.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:24.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:24.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:24.052 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:11:24.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:11:24.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:24.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:24.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:25.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:25.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:25.804 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:11:25.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:11:25.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:25.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:26.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:27.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:27.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:27.246 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:11:27.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:11:27.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:27.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:27.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:28.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:28.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:28.745 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:11:28.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:11:28.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:28.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:11:29.204 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/259082,v1:127.0.0.1:6811/259082] [v2:127.0.0.1:6812/259082,v1:127.0.0.1:6813/259082] exists,up 1150adab-473a-42f5-a409-e1c88618fc18 2024-09-15T03:11:29.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:11:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:11:29.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:11:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=12514411-2a71-4e2f-bd1d-133b314f50cb 2024-09-15T03:11:29.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 12514411-2a71-4e2f-bd1d-133b314f50cb' 2024-09-15T03:11:29.212 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 12514411-2a71-4e2f-bd1d-133b314f50cb 2024-09-15T03:11:29.212 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:11:29.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhUOZmYYJHDRAAKjE2tBxj0+HKK7dujQXIqQ== 2024-09-15T03:11:29.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhUOZmYYJHDRAAKjE2tBxj0+HKK7dujQXIqQ=="}' 2024-09-15T03:11:29.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 12514411-2a71-4e2f-bd1d-133b314f50cb -i td/test-erasure-eio/2/new.json 2024-09-15T03:11:29.840 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:11:29.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T03:11:29.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhUOZmYYJHDRAAKjE2tBxj0+HKK7dujQXIqQ== --osd-uuid 12514411-2a71-4e2f-bd1d-133b314f50cb 2024-09-15T03:11:29.876 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:29.875+0000 7fa50b71e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:29.878 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:29.877+0000 7fa50b71e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:29.878+0000 7fa50b71e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:29.878+0000 7fa50b71e440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:11:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:29.878+0000 7fa50b71e440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:11:29.879 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:29.878+0000 7fa50b71e440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:11:29.955 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:29.954+0000 7fa50b71e440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T03:11:35.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T03:11:35.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:11:35.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:11:35.754 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T03:11:35.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:11:36.236 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:11:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:11:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:11:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:11:36.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:11:36.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:11:36.259 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:36.258+0000 7f37fcc08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:36.261 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:36.260+0000 7f37fcc08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:36.262 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:36.261+0000 7f37fcc08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:11:36.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:11:36.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:11:36.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:11:36.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:11:36.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:11:36.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:36.584 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:11:36.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:11:36.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:36.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:36.776 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:36.775+0000 7f37fcc08440 -1 Falling back to public interface 2024-09-15T03:11:36.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:37.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:37.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:37.934 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:11:37.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:11:37.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:37.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:38.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:39.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:39.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:39.284 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:11:39.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:11:39.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:39.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:39.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:11:39.299+0000 7f37fcc08440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:11:39.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:40.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:40.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:40.752 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:11:40.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:11:40.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:40.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:41.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:42.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:42.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:42.223 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:11:42.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:11:42.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:42.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:42.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:43.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:43.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:43.606 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:11:43.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:11:43.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:43.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:44.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:45.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:45.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:45.072 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:11:45.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:11:45.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:45.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:45.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:46.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:46.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:46.460 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:11:46.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:11:46.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:46.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:46.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:47.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:47.820 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:11:47.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:47.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:11:47.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:47.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:48.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:49.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:49.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:49.190 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:11:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:11:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:49.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:49.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:50.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:50.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:50.613 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:11:50.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:11:50.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:50.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:50.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:51.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:51.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:51.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:11:51.971 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:11:51.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:51.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:52.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:53.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:53.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:53.653 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:11:53.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:11:53.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:53.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:54.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:55.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:55.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:55.448 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:11:55.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:11:55.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:55.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:56.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:57.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:57.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:57.293 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:11:57.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:11:57.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:57.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:57.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:11:58.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:11:58.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:11:58.890 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:11:58.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:11:58.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:11:58.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:11:59.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:00.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:00.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:00.555 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:12:00.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:12:00.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:00.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:12:01.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:02.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:02.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:02.384 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:12:02.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:12:02.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:02.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:12:02.800 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/260836,v1:127.0.0.1:6819/260836] [v2:127.0.0.1:6820/260836,v1:127.0.0.1:6821/260836] exists,up 12514411-2a71-4e2f-bd1d-133b314f50cb 2024-09-15T03:12:02.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:12:02.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:12:02.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:12:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:12:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T03:12:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:12:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:12:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T03:12:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:12:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T03:12:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:12:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:12:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:12:02.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:12:02.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T03:12:02.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T03:12:02.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:12:02.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:12:02.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:12:02.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:12:02.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:12:02.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:12:02.812 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:12:02.812 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:02.813 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:12:02.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:12:02.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:12:02.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:12:02.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:12:02.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:12:02.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:12:02.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:12:02.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:12:02.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:12:02.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:12:02.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:12:02.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:12:02.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:12:02.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T03:12:02.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:12:02.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=217fbb2d-77e2-4f9e-8ea8-d76d45ccad40 2024-09-15T03:12:02.816 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 217fbb2d-77e2-4f9e-8ea8-d76d45ccad40 2024-09-15T03:12:02.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 217fbb2d-77e2-4f9e-8ea8-d76d45ccad40' 2024-09-15T03:12:02.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:12:02.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCUOZm98TAMBAAEYpb6f2V2b85acIPPlOTkQ== 2024-09-15T03:12:02.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCUOZm98TAMBAAEYpb6f2V2b85acIPPlOTkQ=="}' 2024-09-15T03:12:02.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 217fbb2d-77e2-4f9e-8ea8-d76d45ccad40 -i td/test-erasure-eio/3/new.json 2024-09-15T03:12:03.989 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:12:04.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T03:12:04.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCUOZm98TAMBAAEYpb6f2V2b85acIPPlOTkQ== --osd-uuid 217fbb2d-77e2-4f9e-8ea8-d76d45ccad40 2024-09-15T03:12:04.026 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:04.025+0000 7fc96060a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:04.069 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:04.067+0000 7fc96060a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:04.069 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:04.068+0000 7fc96060a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:04.070 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:04.069+0000 7fc96060a440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:12:04.071 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:04.069+0000 7fc96060a440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:12:04.071 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:04.069+0000 7fc96060a440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:12:04.203 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:04.202+0000 7fc96060a440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T03:12:09.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T03:12:09.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:12:09.807 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T03:12:09.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T03:12:09.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:12:10.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T03:12:10.315 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T03:12:10.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:12:10.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:12:10.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:12:10.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:12:10.337 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:10.336+0000 7f81b1a4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:10.339 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:10.338+0000 7f81b1a4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:10.340 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:10.339+0000 7f81b1a4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:10.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T03:12:10.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:12:10.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T03:12:10.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:12:10.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:12:10.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:10.663 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:12:10.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:12:10.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:10.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:10.862 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:10.861+0000 7f81b1a4f440 -1 Falling back to public interface 2024-09-15T03:12:11.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:12.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:12.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:12.025 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:12:12.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:12:12.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:12.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:12.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:13.322 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:13.321+0000 7f81b1a4f440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:12:13.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:13.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:12:13.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:12:13.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:13.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:13.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:14.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:14.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:14.891 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:12:14.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:12:14.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:14.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:15.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:16.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:16.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:16.743 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:12:16.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:12:16.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:16.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:17.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:18.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:18.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:18.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:12:18.207 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:12:18.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:18.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:18.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:19.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:19.614 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:12:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:12:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:19.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:20.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:21.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:21.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:21.088 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:12:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:12:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:21.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:21.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:22.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:22.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:22.461 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:12:22.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:12:22.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:22.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:22.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:23.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:23.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:23.838 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:12:23.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:12:23.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:23.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:24.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:25.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:25.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:25.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:12:25.239 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:12:25.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:25.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:25.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:26.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:26.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:26.774 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:12:26.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:12:26.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:26.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:27.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:28.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:28.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:28.466 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:12:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:12:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:28.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:29.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:30.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:30.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:30.210 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:12:30.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:12:30.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:30.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:31.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:32.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:32.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:32.194 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:12:32.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:12:32.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:32.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:33.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:34.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:34.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:34.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:12:34.291 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:12:34.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:34.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:34.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:35.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:35.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:35.856 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:12:35.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:12:35.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:35.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:36.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:37.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:37.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:37.721 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:12:37.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:12:37.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:37.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:12:38.104 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/262590,v1:127.0.0.1:6827/262590] [v2:127.0.0.1:6828/262590,v1:127.0.0.1:6829/262590] exists,up 217fbb2d-77e2-4f9e-8ea8-d76d45ccad40 2024-09-15T03:12:38.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:12:38.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:12:38.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:12:38.105 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T03:12:38.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T03:12:38.178 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:12:38.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T03:12:38.188 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:10:28.595+0000 7f45c4e2d440 0 load: jerasure load: lrc 2024-09-15T03:12:38.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:285: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2024-09-15T03:12:38.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:286: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T03:12:38.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:12:38.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T03:12:38.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T03:12:38.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T03:12:38.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T03:12:38.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T03:12:38.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T03:12:39.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T03:12:39.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T03:12:40.127 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T03:12:40.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:12:41.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T03:12:41.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:12:41.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:12:41.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:12:41.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:12:41.141 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:12:41.141 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:12:41.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:12:41.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:12:41.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:12:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:12:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:12:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:12:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:12:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:12:41.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:12:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:12:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:12:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:12:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:12:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:12:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:41.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:12:41.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803801 2024-09-15T03:12:41.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803801 2024-09-15T03:12:41.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803801' 2024-09-15T03:12:41.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:41.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:12:41.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542168 2024-09-15T03:12:41.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542168 2024-09-15T03:12:41.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803801 1-60129542168' 2024-09-15T03:12:41.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:41.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:12:41.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345934 2024-09-15T03:12:41.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345934 2024-09-15T03:12:41.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803801 1-60129542168 2-85899345934' 2024-09-15T03:12:41.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:41.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:12:42.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=133143986183 2024-09-15T03:12:42.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 133143986183 2024-09-15T03:12:42.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803801 1-60129542168 2-85899345934 3-133143986183' 2024-09-15T03:12:42.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:42.085 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803801 2024-09-15T03:12:42.085 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:42.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:12:42.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803801 2024-09-15T03:12:42.086 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:42.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803801 2024-09-15T03:12:42.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2024-09-15T03:12:42.087 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803801 2024-09-15T03:12:42.088 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:12:42.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803799 -lt 25769803801 2024-09-15T03:12:42.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:12:43.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:12:43.434 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:12:43.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803801 -lt 25769803801 2024-09-15T03:12:43.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:43.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542168 2024-09-15T03:12:43.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:43.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:12:43.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542168 2024-09-15T03:12:43.791 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:43.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542168 2024-09-15T03:12:43.792 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 60129542168 2024-09-15T03:12:43.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542168' 2024-09-15T03:12:43.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:12:44.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542168 -lt 60129542168 2024-09-15T03:12:44.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:44.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345934 2024-09-15T03:12:44.163 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:44.163 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:12:44.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345934 2024-09-15T03:12:44.164 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:44.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345934 2024-09-15T03:12:44.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345934' 2024-09-15T03:12:44.166 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 85899345934 2024-09-15T03:12:44.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:12:44.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345935 -lt 85899345934 2024-09-15T03:12:44.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:44.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-133143986183 2024-09-15T03:12:44.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:44.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:12:44.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-133143986183 2024-09-15T03:12:44.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:44.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=133143986183 2024-09-15T03:12:44.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 133143986183' 2024-09-15T03:12:44.530 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 133143986183 2024-09-15T03:12:44.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:12:44.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 133143986184 -lt 133143986183 2024-09-15T03:12:44.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:12:44.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:44.896 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:45.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:12:45.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:12:45.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:12:45.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:12:45.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:12:45.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:12:45.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:12:45.370 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:12:45.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:12:45.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:12:45.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:45.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:46.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:12:46.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:12:46.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_missing: local shard_id=1 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:289: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-87967-1 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-87967-1 2024-09-15T03:12:46.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-87967-1 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:12:46.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-87967-1 td/test-erasure-eio/ORIGINAL 2024-09-15T03:12:46.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-87967-1 td/test-erasure-eio 1 2024-09-15T03:12:46.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-15T03:12:46.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-15T03:12:46.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-15T03:12:46.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-87967-1 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-87967-1 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:12:46.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-87967-1 2024-09-15T03:12:46.370 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-87967-1 2024-09-15T03:12:46.370 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-87967-1 remove 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:12:46.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-87967-1 remove 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:12:46.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:12:46.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:12:46.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-87967-1 remove 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-15T03:12:47.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-87967-1 remove 2024-09-15T03:12:48.947 INFO:tasks.workunit.client.0.smithi130.stdout:remove 1#2:7f2b2561:::obj-remove-87967-1:head# 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:12:49.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:12:49.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:12:49.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:12:49.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:12:49.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:12:49.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:12:49.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:12:49.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:12:49.469 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.1 2024-09-15T03:12:49.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:12:49.470 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-15T03:12:49.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:12:49.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:12:49.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:12:49.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:12:49.491 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:49.489+0000 7f6b69d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:49.493 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:49.491+0000 7f6b69d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:49.494 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:49.492+0000 7f6b69d2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:12:49.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:12:49.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:49.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:12:50.010 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:50.008+0000 7f6b69d2f440 -1 Falling back to public interface 2024-09-15T03:12:50.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:12:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:12:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:51.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:12:51.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:52.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:52.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:52.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:12:52.543 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:12:52.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:52.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:12:52.640 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:12:52.638+0000 7f6b69d2f440 -1 osd.1 39 log_to_monitors true 2024-09-15T03:12:53.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:12:54.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:12:54.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:12:54.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:12:54.089 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:12:54.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:12:54.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:12:54.638 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 40 last_clean_interval [14,39) [v2:127.0.0.1:6810/264394,v1:127.0.0.1:6811/264394] [v2:127.0.0.1:6812/264394,v1:127.0.0.1:6813/264394] exists,up 1150adab-473a-42f5-a409-e1c88618fc18 2024-09-15T03:12:54.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:12:54.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:12:54.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:12:54.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:12:54.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:12:54.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:12:54.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:12:54.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:12:54.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:12:54.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:12:54.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:12:54.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:12:55.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:12:55.079 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:12:55.080 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:12:55.080 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:12:55.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:12:55.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:55.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:12:55.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803806 2024-09-15T03:12:55.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803806 2024-09-15T03:12:55.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806' 2024-09-15T03:12:55.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:55.187 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:12:55.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561027 2024-09-15T03:12:55.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561027 2024-09-15T03:12:55.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-188978561027' 2024-09-15T03:12:55.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:55.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:12:55.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345939 2024-09-15T03:12:55.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345939 2024-09-15T03:12:55.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-188978561027 2-85899345939' 2024-09-15T03:12:55.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:12:55.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:12:55.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=133143986188 2024-09-15T03:12:55.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 133143986188 2024-09-15T03:12:55.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803806 1-188978561027 2-85899345939 3-133143986188' 2024-09-15T03:12:55.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:55.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803806 2024-09-15T03:12:55.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:55.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:12:55.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803806 2024-09-15T03:12:55.505 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:55.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803806 2024-09-15T03:12:55.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803806' 2024-09-15T03:12:55.507 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803806 2024-09-15T03:12:55.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:12:55.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803806 -lt 25769803806 2024-09-15T03:12:55.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:55.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-188978561027 2024-09-15T03:12:55.855 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:55.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:12:55.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-188978561027 2024-09-15T03:12:55.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:55.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561027 2024-09-15T03:12:55.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-09-15T03:12:55.857 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 188978561027 2024-09-15T03:12:55.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:12:56.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561027 -lt 188978561027 2024-09-15T03:12:56.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:56.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345939 2024-09-15T03:12:56.309 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:56.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:12:56.310 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345939 2024-09-15T03:12:56.311 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:56.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345939 2024-09-15T03:12:56.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345939' 2024-09-15T03:12:56.312 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 85899345939 2024-09-15T03:12:56.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:12:56.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345938 -lt 85899345939 2024-09-15T03:12:56.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:12:57.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:12:57.660 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:12:58.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-15T03:12:58.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:12:58.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-133143986188 2024-09-15T03:12:58.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:12:58.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:12:58.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-133143986188 2024-09-15T03:12:58.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:12:58.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=133143986188 2024-09-15T03:12:58.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 133143986188' 2024-09-15T03:12:58.033 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 133143986188 2024-09-15T03:12:58.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:12:58.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 133143986188 -lt 133143986188 2024-09-15T03:12:58.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:12:58.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:58.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:58.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:12:58.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:12:58.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:12:58.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:12:58.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:12:58.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:12:58.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:12:58.908 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:12:59.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:12:59.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:12:59.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:12:59.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-87967-1 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-87967-1 2024-09-15T03:12:59.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:12:59.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:12:59.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-87967-1 td/test-erasure-eio/COPY 2024-09-15T03:12:59.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:12:59.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:12:59.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-15T03:12:59.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-15T03:12:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-09-15T03:12:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-87967-1 td/test-erasure-eio 2 2024-09-15T03:12:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-87967-1 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-87967-1 2024-09-15T03:12:59.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:12:59.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-87967-1 2024-09-15T03:12:59.809 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-87967-1 2024-09-15T03:12:59.809 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:13:00.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:13:00.160 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:13:00.160 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:13:00.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:13:00.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-09-15T03:13:00.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-15T03:13:00.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2024-09-15T03:13:00.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-87967-1 remove 2024-09-15T03:13:00.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-15T03:13:00.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T03:13:00.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-15T03:13:00.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T03:13:00.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-87967-1 remove 2024-09-15T03:13:00.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:13:00.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:13:00.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:13:00.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-87967-1 remove 2024-09-15T03:13:00.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-15T03:13:00.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:13:00.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-15T03:13:00.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:13:00.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-15T03:13:00.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-87967-1 remove 2024-09-15T03:13:02.599 INFO:tasks.workunit.client.0.smithi130.stdout:remove 2#2:7f2b2561:::obj-remove-87967-1:head# 2024-09-15T03:13:03.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:13:03.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:03.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:13:03.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:13:03.120 INFO:tasks.workunit.client.0.smithi130.stderr:start osd.2 2024-09-15T03:13:03.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:13:03.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T03:13:03.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:13:03.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:13:03.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:13:03.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:13:03.143 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:03.142+0000 7f72cb929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:03.145 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:03.144+0000 7f72cb929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:03.146 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:03.145+0000 7f72cb929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:03.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:13:03.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:13:03.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:13:03.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:13:03.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:13:03.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:03.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:13:03.477 INFO:tasks.workunit.client.0.smithi130.stderr:0 2024-09-15T03:13:03.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:03.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:13:03.664 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:03.663+0000 7f72cb929440 -1 Falling back to public interface 2024-09-15T03:13:03.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:04.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:04.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:04.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:13:04.827 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:13:04.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:04.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:13:05.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:06.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:06.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:06.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:13:06.187 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:13:06.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:06.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:13:06.317 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:06.316+0000 7f72cb929440 -1 osd.2 46 log_to_monitors true 2024-09-15T03:13:06.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:07.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:07.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:07.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:13:07.633 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-15T03:13:07.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:07.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:13:08.157 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 up in weight 1 up_from 51 up_thru 40 down_at 47 last_clean_interval [20,46) [v2:127.0.0.1:6818/265800,v1:127.0.0.1:6819/265800] [v2:127.0.0.1:6820/265800,v1:127.0.0.1:6821/265800] exists,up 12514411-2a71-4e2f-bd1d-133b314f50cb 2024-09-15T03:13:08.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:13:08.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:13:08.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:13:08.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:13:08.158 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:13:08.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:13:08.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:13:08.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:13:08.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:13:08.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:13:08.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:13:08.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:13:08.640 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:13:08.641 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:13:08.641 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:13:08.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:13:08.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:13:08.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:13:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803810 2024-09-15T03:13:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803810 2024-09-15T03:13:08.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810' 2024-09-15T03:13:08.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:13:08.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:13:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561032 2024-09-15T03:13:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561032 2024-09-15T03:13:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-188978561032' 2024-09-15T03:13:08.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:13:08.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:13:08.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332099 2024-09-15T03:13:08.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332099 2024-09-15T03:13:08.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-188978561032 2-219043332099' 2024-09-15T03:13:08.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:13:08.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:13:09.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=133143986193 2024-09-15T03:13:09.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 133143986193 2024-09-15T03:13:09.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-188978561032 2-219043332099 3-133143986193' 2024-09-15T03:13:09.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:13:09.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803810 2024-09-15T03:13:09.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:13:09.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:13:09.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803810 2024-09-15T03:13:09.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:13:09.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803810 2024-09-15T03:13:09.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-09-15T03:13:09.096 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.0 seq 25769803810 2024-09-15T03:13:09.096 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:13:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803810 2024-09-15T03:13:09.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:13:10.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:13:10.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:13:10.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803810 2024-09-15T03:13:10.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:13:10.800 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-188978561032 2024-09-15T03:13:10.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:13:10.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:13:10.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-188978561032 2024-09-15T03:13:10.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:13:10.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561032 2024-09-15T03:13:10.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 188978561032' 2024-09-15T03:13:10.803 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.1 seq 188978561032 2024-09-15T03:13:10.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:13:11.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561032 -lt 188978561032 2024-09-15T03:13:11.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:13:11.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-219043332099 2024-09-15T03:13:11.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:13:11.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:13:11.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-219043332099 2024-09-15T03:13:11.155 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:13:11.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332099 2024-09-15T03:13:11.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 219043332099' 2024-09-15T03:13:11.156 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.2 seq 219043332099 2024-09-15T03:13:11.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:13:11.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332099 -lt 219043332099 2024-09-15T03:13:11.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:13:11.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-133143986193 2024-09-15T03:13:11.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:13:11.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:13:11.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-133143986193 2024-09-15T03:13:11.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:13:11.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=133143986193 2024-09-15T03:13:11.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 133143986193' 2024-09-15T03:13:11.517 INFO:tasks.workunit.client.0.smithi130.stderr:waiting osd.3 seq 133143986193 2024-09-15T03:13:11.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:13:11.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 133143986193 -lt 133143986193 2024-09-15T03:13:11.868 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:13:11.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:13:11.869 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:13:12.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:13:12.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:13:12.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:13:12.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:13:12.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:13:12.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:13:12.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:13:12.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:13:12.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:13:12.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:13:12.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:13:12.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:13:13.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-87967-1 fail 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-87967-1 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-15T03:13:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-remove-87967-1 td/test-erasure-eio/COPY 2024-09-15T03:13:13.204 INFO:tasks.workunit.client.0.smithi130.stderr:error getting pool-jerasure/obj-remove-87967-1: (5) Input/output error 2024-09-15T03:13:13.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-15T03:13:13.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-15T03:13:13.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2024-09-15T03:13:13.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:13:13.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T03:13:13.790 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T03:13:13.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T03:13:14.423 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T03:13:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T03:13:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:13:14.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:13:14.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:13:14.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:13:14.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:13:14.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:13:14.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:13:14.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:13:14.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:13:14.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:13:14.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:13:14.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:13:14.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:13:14.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:13:14.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:13:14.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:14.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:13:14.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:13:14.548 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:14.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:13:14.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:13:14.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T03:13:14.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:13:14.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:13:14.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:13:14.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:13:14.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:13:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:13:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:13:14.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:13:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:13:14.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:13:14.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:13:14.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:13:14.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:13:14.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:13:14.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:13:14.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:14.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:14.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:13:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:13:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:13:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T03:13:14.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:13:14.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:14.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T03:13:14.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:13:14.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:13:14.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:13:14.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T03:13:14.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T03:13:14.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T03:13:14.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:13:14.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:13:14.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:13:14.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T03:13:14.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T03:13:14.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:13:14.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:14.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:14.923 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:14.923 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:14.923 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:14.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:14.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:13:15.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:13:15.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:13:15.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:13:15.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:13:15.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:13:15.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:13:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:13:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:13:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:13:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:13:15.175 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:15.176 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:15.176 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:13:15.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:13:15.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T03:13:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:13:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:13:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:13:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:13:15.250 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:13:15.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:13:15.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:13:15.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:13:15.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:13:15.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:15.251 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:15.251 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:13:15.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:13:15.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T03:13:15.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T03:13:15.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T03:13:15.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:13:15.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:13:15.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:13:15.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T03:13:15.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:16.159 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:13:16.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:13:16.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T03:13:16.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:13:16.540 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T03:13:16.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:13:17.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T03:13:17.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:13:17.552 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:13:17.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:13:17.552 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:17.553 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:17.553 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:13:17.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T03:13:17.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T03:13:17.619 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:13:17.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T03:13:17.628 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:13:14.949+0000 7f1dcda6e900 0 load: jerasure load: lrc 2024-09-15T03:13:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2024-09-15T03:13:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2024-09-15T03:13:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2024-09-15T03:13:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:333: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2024-09-15T03:13:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T03:13:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T03:13:17.629 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T03:13:17.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:13:17.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:17.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T03:13:17.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:13:17.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d49c9010-6e1f-4f94-9014-60437c0f7877 2024-09-15T03:13:17.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d49c9010-6e1f-4f94-9014-60437c0f7877' 2024-09-15T03:13:17.637 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 d49c9010-6e1f-4f94-9014-60437c0f7877 2024-09-15T03:13:17.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:13:17.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNUOZmIBGgJhAAGfcS4OmLbufG2K6zHmM19Q== 2024-09-15T03:13:17.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNUOZmIBGgJhAAGfcS4OmLbufG2K6zHmM19Q=="}' 2024-09-15T03:13:17.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d49c9010-6e1f-4f94-9014-60437c0f7877 -i td/test-erasure-eio/0/new.json 2024-09-15T03:13:17.966 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:13:17.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T03:13:17.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNUOZmIBGgJhAAGfcS4OmLbufG2K6zHmM19Q== --osd-uuid d49c9010-6e1f-4f94-9014-60437c0f7877 2024-09-15T03:13:18.001 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:18.000+0000 7f944ad25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:18.003 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:18.002+0000 7f944ad25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:18.004 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:18.003+0000 7f944ad25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:18.005 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:18.004+0000 7f944ad25440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:13:18.005 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:18.004+0000 7f944ad25440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:13:18.005 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:18.004+0000 7f944ad25440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:13:18.094 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:18.093+0000 7f944ad25440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T03:13:24.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T03:13:24.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:13:24.148 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T03:13:24.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:13:24.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:13:24.641 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T03:13:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:13:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:13:24.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:13:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:13:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:13:24.664 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:24.663+0000 7fb753812440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:24.666 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:24.665+0000 7fb753812440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:24.667 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:24.666+0000 7fb753812440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:24.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:13:24.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:13:24.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:13:24.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:13:24.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:13:24.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:24.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:13:24.991 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:13:24.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:24.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:25.191 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:25.190+0000 7fb753812440 -1 Falling back to public interface 2024-09-15T03:13:25.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:26.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:26.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:26.339 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:13:26.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:13:26.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:26.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:26.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:27.657 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:27.655+0000 7fb753812440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:13:27.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:27.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:27.699 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:13:27.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:13:27.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:27.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:28.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:29.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:29.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:29.217 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:13:29.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:13:29.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:29.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:29.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:30.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:30.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:30.606 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:13:30.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:13:30.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:30.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:30.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:31.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:31.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:31.974 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:13:31.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:13:31.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:31.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:32.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:33.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:33.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:33.498 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:13:33.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:13:33.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:33.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:33.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:34.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:34.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:34.878 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:13:34.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:13:34.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:34.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:35.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:36.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:36.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:36.333 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:13:36.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:13:36.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:36.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:36.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:37.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:37.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:37.692 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:13:37.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:13:37.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:37.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:38.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:39.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:39.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:39.076 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:13:39.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:13:39.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:39.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:39.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:40.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:40.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:40.839 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:13:40.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:13:40.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:40.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:42.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:42.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:42.682 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:13:42.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:13:42.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:42.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:43.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:44.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:44.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:44.570 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:13:44.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:13:44.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:44.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:45.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:46.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:46.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:46.379 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:13:46.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:13:46.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:46.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:47.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:48.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:48.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:48.277 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:13:48.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:13:48.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:48.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:49.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:13:50.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:13:50.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:50.107 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:13:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:13:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:50.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:13:50.593 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/267968,v1:127.0.0.1:6803/267968] [v2:127.0.0.1:6804/267968,v1:127.0.0.1:6805/267968] exists,up d49c9010-6e1f-4f94-9014-60437c0f7877 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:13:50.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:13:50.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:13:50.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:13:50.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:13:50.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:13:50.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:13:50.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:13:50.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:13:50.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=80b52d55-9d49-4cd7-8308-3112b23b8609 2024-09-15T03:13:50.600 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 80b52d55-9d49-4cd7-8308-3112b23b8609 2024-09-15T03:13:50.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 80b52d55-9d49-4cd7-8308-3112b23b8609' 2024-09-15T03:13:50.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:13:50.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuUOZmGcBxJBAA+oZRYZhFke21+YNfDwwvfw== 2024-09-15T03:13:50.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuUOZmGcBxJBAA+oZRYZhFke21+YNfDwwvfw=="}' 2024-09-15T03:13:50.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 80b52d55-9d49-4cd7-8308-3112b23b8609 -i td/test-erasure-eio/1/new.json 2024-09-15T03:13:51.478 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:13:51.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T03:13:51.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuUOZmGcBxJBAA+oZRYZhFke21+YNfDwwvfw== --osd-uuid 80b52d55-9d49-4cd7-8308-3112b23b8609 2024-09-15T03:13:51.515 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:51.514+0000 7f323ec15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:51.537 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:51.536+0000 7f323ec15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:51.538 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:51.537+0000 7f323ec15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:51.538 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:51.537+0000 7f323ec15440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:13:51.538 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:51.537+0000 7f323ec15440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:13:51.539 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:51.537+0000 7f323ec15440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:13:51.612 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:51.610+0000 7f323ec15440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T03:13:57.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T03:13:57.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:13:57.766 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T03:13:57.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:13:57.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:13:58.315 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T03:13:58.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:13:58.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:13:58.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:13:58.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:13:58.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:13:58.338 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:58.337+0000 7fb92ce15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:58.340 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:58.339+0000 7fb92ce15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:58.341 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:58.340+0000 7fb92ce15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:13:58.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:13:58.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:13:58.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:13:58.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:13:58.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:13:58.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:13:58.665 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:13:58.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:13:58.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:13:58.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:13:58.855 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:13:58.854+0000 7fb92ce15440 -1 Falling back to public interface 2024-09-15T03:13:59.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:00.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:00.012 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:14:00.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:14:00.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:00.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:00.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:01.345 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:01.344+0000 7fb92ce15440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:14:01.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:01.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:01.377 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:14:01.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:14:01.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:01.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:02.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:03.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:03.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:03.083 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:14:03.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:14:03.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:03.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:03.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:04.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:04.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:04.477 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:14:04.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:14:04.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:04.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:05.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:06.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:06.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:06.085 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:14:06.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:14:06.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:06.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:06.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:07.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:07.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:07.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:14:07.482 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:14:07.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:07.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:07.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:08.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:08.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:08.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:14:08.886 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:14:08.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:08.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:09.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:10.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:10.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:10.294 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:14:10.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:14:10.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:10.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:10.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:11.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:11.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:11.739 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:14:11.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:14:11.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:11.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:12.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:13.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:13.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:13.312 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:14:13.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:14:13.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:13.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:13.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:14.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:14.769 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:14:14.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:14:14.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:14.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:15.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:16.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:16.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:16.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:14:16.635 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:14:16.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:16.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:17.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:18.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:18.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:18.461 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:14:18.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:14:18.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:18.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:19.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:20.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:20.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:20.172 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:14:20.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:14:20.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:20.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:20.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:21.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:21.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:21.984 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:14:21.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:14:21.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:21.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:22.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:23.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:23.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:23.965 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:14:23.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:14:23.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:23.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:24.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:25.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:25.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:25.374 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:14:25.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:14:25.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:25.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:14:25.731 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/269698,v1:127.0.0.1:6811/269698] [v2:127.0.0.1:6812/269698,v1:127.0.0.1:6813/269698] exists,up 80b52d55-9d49-4cd7-8308-3112b23b8609 2024-09-15T03:14:25.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:14:25.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:14:25.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:14:25.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:14:25.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:14:25.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:14:25.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=92b11500-291a-48bd-96e1-7e844ecb61ec 2024-09-15T03:14:25.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 92b11500-291a-48bd-96e1-7e844ecb61ec' 2024-09-15T03:14:25.738 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 92b11500-291a-48bd-96e1-7e844ecb61ec 2024-09-15T03:14:25.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:14:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARUeZmwRqhLBAAxYp+4aRxanMdENZN9rxpnQ== 2024-09-15T03:14:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARUeZmwRqhLBAAxYp+4aRxanMdENZN9rxpnQ=="}' 2024-09-15T03:14:25.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 92b11500-291a-48bd-96e1-7e844ecb61ec -i td/test-erasure-eio/2/new.json 2024-09-15T03:14:26.219 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:14:26.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T03:14:26.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARUeZmwRqhLBAAxYp+4aRxanMdENZN9rxpnQ== --osd-uuid 92b11500-291a-48bd-96e1-7e844ecb61ec 2024-09-15T03:14:26.254 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:26.253+0000 7f488d616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:26.256 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:26.255+0000 7f488d616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:26.257 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:26.256+0000 7f488d616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:26.257 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:26.256+0000 7f488d616440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:14:26.257 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:26.256+0000 7f488d616440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:14:26.257 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:26.256+0000 7f488d616440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:14:26.334 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:26.333+0000 7f488d616440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T03:14:32.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T03:14:32.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:14:32.165 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T03:14:32.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:14:32.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:14:32.665 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:14:32.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:14:32.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:14:32.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:14:32.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:14:32.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:14:32.687 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:32.686+0000 7f1a5f669440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:32.689 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:32.688+0000 7f1a5f669440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:32.690 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:32.689+0000 7f1a5f669440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:14:33.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:14:33.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:14:33.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:14:33.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:14:33.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:14:33.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:33.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:14:33.018 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:14:33.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:33.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:33.208 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:33.207+0000 7f1a5f669440 -1 Falling back to public interface 2024-09-15T03:14:33.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:34.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:34.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:34.369 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:14:34.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:14:34.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:34.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:34.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:35.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:35.726 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:14:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:14:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:35.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:35.752 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:14:35.751+0000 7f1a5f669440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:14:36.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:37.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:37.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:37.267 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:14:37.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:14:37.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:37.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:37.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:38.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:38.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:38.710 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:14:38.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:14:38.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:38.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:39.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:40.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:40.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:40.068 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:14:40.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:14:40.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:40.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:40.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:41.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:41.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:41.498 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:14:41.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:14:41.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:41.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:41.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:42.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:42.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:42.859 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:14:42.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:14:42.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:42.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:43.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:44.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:44.245 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:14:44.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:44.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:14:44.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:44.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:44.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:45.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:45.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:45.611 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:14:45.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:14:45.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:45.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:45.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:46.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:46.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:46.987 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:14:46.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:14:46.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:46.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:47.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:48.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:48.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:48.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:14:48.372 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:14:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:48.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:49.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:50.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:50.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:50.363 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:14:50.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:14:50.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:50.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:51.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:52.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:52.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:52.129 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:14:52.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:14:52.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:52.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:53.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:53.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:53.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:14:53.940 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:14:53.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:53.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:54.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:55.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:55.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:55.446 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:14:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:14:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:56.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:57.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:57.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:57.136 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:14:57.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:14:57.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:57.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:57.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:14:58.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:14:58.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:14:58.872 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:14:58.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:14:58.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:14:58.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:14:59.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:00.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:00.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:00.335 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T03:15:00.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:15:00.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:00.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:00.809 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/271453,v1:127.0.0.1:6819/271453] [v2:127.0.0.1:6820/271453,v1:127.0.0.1:6821/271453] exists,up 92b11500-291a-48bd-96e1-7e844ecb61ec 2024-09-15T03:15:00.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:15:00.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:15:00.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T03:15:00.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:15:00.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:15:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=81ab5797-61dc-4e0d-968e-d35e59eda8f8 2024-09-15T03:15:00.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 81ab5797-61dc-4e0d-968e-d35e59eda8f8' 2024-09-15T03:15:00.817 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 81ab5797-61dc-4e0d-968e-d35e59eda8f8 2024-09-15T03:15:00.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:15:00.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0UeZm9U1KMRAAb4JqAwKUST37j/KZHdPalw== 2024-09-15T03:15:00.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0UeZm9U1KMRAAb4JqAwKUST37j/KZHdPalw=="}' 2024-09-15T03:15:00.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 81ab5797-61dc-4e0d-968e-d35e59eda8f8 -i td/test-erasure-eio/3/new.json 2024-09-15T03:15:01.359 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:15:01.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T03:15:01.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0UeZm9U1KMRAAb4JqAwKUST37j/KZHdPalw== --osd-uuid 81ab5797-61dc-4e0d-968e-d35e59eda8f8 2024-09-15T03:15:01.395 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:01.394+0000 7fe656c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:01.397 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:01.396+0000 7fe656c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:01.398 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:01.397+0000 7fe656c08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:01.398 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:01.397+0000 7fe656c08440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:15:01.398 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:01.397+0000 7fe656c08440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:15:01.398 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:01.397+0000 7fe656c08440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:15:01.474 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:01.473+0000 7fe656c08440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T03:15:07.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T03:15:07.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:15:07.393 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T03:15:07.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T03:15:07.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:15:07.898 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T03:15:07.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T03:15:07.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:15:07.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:15:07.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:15:07.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:15:07.921 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:07.920+0000 7f297c551440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:07.946 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:07.945+0000 7f297c551440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:07.947 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:07.947+0000 7f297c551440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:08.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T03:15:08.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:15:08.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T03:15:08.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:15:08.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:15:08.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:08.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:15:08.336 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:15:08.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:08.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:08.482 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:08.481+0000 7f297c551440 -1 Falling back to public interface 2024-09-15T03:15:08.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:09.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:09.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:09.684 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:15:09.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:15:09.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:09.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:10.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:11.017 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:11.016+0000 7f297c551440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:15:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:15:11.061 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:15:11.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:11.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:11.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:12.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:12.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:12.573 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:15:12.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:15:12.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:12.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:12.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:13.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:13.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:13.975 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:15:13.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:15:13.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:13.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:14.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:15.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:15.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:15.433 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:15:15.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:15:15.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:15.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:15.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:16.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:16.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:16.822 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:15:16.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:15:16.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:16.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:17.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:18.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:18.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:18.206 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:15:18.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:15:18.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:18.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:18.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:19.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:19.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:19.666 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:15:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:15:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:20.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:21.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:21.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:21.113 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:15:21.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:15:21.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:21.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:22.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:22.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:22.518 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:15:22.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:15:22.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:22.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:23.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:24.014 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:15:24.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:15:24.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:24.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:24.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:25.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:25.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:25.585 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:15:25.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:15:25.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:25.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:26.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:27.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:27.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:27.072 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:15:27.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:15:27.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:27.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:27.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:28.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:28.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:28.785 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:15:28.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:15:28.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:28.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:29.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:30.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:30.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:30.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:15:30.635 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:15:30.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:30.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:31.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:32.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:32.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:32.774 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:15:32.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:15:32.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:32.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:33.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:34.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:34.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:34.545 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:15:34.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:15:34.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:34.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:35.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:36.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:36.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:36.131 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T03:15:36.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:15:36.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:36.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:15:36.517 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/273233,v1:127.0.0.1:6827/273233] [v2:127.0.0.1:6828/273233,v1:127.0.0.1:6829/273233] exists,up 81ab5797-61dc-4e0d-968e-d35e59eda8f8 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:15:36.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T03:15:36.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T03:15:36.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T03:15:36.593 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:15:36.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T03:15:36.602 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:13:25.194+0000 7fb753812440 0 load: jerasure load: lrc 2024-09-15T03:15:36.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:335: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2024-09-15T03:15:36.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T03:15:36.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:15:36.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T03:15:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T03:15:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T03:15:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T03:15:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T03:15:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T03:15:37.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T03:15:37.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T03:15:39.138 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T03:15:39.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:15:40.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T03:15:40.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:15:40.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:15:40.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:15:40.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:15:40.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:15:40.152 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:15:40.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:15:40.152 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:15:40.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:15:40.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:15:40.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:15:40.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:15:40.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:15:40.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:15:40.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:15:40.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:15:40.626 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:15:40.626 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:15:40.626 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:15:40.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:15:40.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:40.627 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:15:40.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803803 2024-09-15T03:15:40.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803803 2024-09-15T03:15:40.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803' 2024-09-15T03:15:40.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:40.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:15:40.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574866 2024-09-15T03:15:40.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574866 2024-09-15T03:15:40.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574866' 2024-09-15T03:15:40.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:40.838 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:15:40.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378640 2024-09-15T03:15:40.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378640 2024-09-15T03:15:40.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574866 2-81604378640' 2024-09-15T03:15:40.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:40.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:15:41.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084294 2024-09-15T03:15:41.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084294 2024-09-15T03:15:41.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574866 2-81604378640 3-120259084294' 2024-09-15T03:15:41.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:41.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803803 2024-09-15T03:15:41.106 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:41.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:15:41.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803803 2024-09-15T03:15:41.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:41.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803803 2024-09-15T03:15:41.108 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803803 2024-09-15T03:15:41.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-09-15T03:15:41.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:15:41.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803801 -lt 25769803803 2024-09-15T03:15:41.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:15:42.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:15:42.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:15:42.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803804 -lt 25769803803 2024-09-15T03:15:42.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:42.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574866 2024-09-15T03:15:42.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:42.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:15:42.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574866 2024-09-15T03:15:42.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:42.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574866 2024-09-15T03:15:42.823 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574866 2024-09-15T03:15:42.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574866' 2024-09-15T03:15:42.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:15:43.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574866 -lt 55834574866 2024-09-15T03:15:43.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:43.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378640 2024-09-15T03:15:43.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:43.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:15:43.191 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378640 2024-09-15T03:15:43.192 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:43.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378640 2024-09-15T03:15:43.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378640' 2024-09-15T03:15:43.193 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378640 2024-09-15T03:15:43.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:15:43.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378640 -lt 81604378640 2024-09-15T03:15:43.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:43.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084294 2024-09-15T03:15:43.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:43.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:15:43.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084294 2024-09-15T03:15:43.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:43.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084294 2024-09-15T03:15:43.545 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084294 2024-09-15T03:15:43.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084294' 2024-09-15T03:15:43.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:15:43.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084295 -lt 120259084294 2024-09-15T03:15:43.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:15:43.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:15:43.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:15:44.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:15:44.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:15:44.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:15:44.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:15:44.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:15:44.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:15:44.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:15:44.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:15:44.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:15:44.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:15:44.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:15:44.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:15:45.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:15:45.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:15:45.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:15:45.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:338: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87967-0 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87967-0 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:15:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87967-0 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:15:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87967-0 td/test-erasure-eio/ORIGINAL 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87967-0 td/test-erasure-eio 0 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-87967-0 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T03:15:45.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T03:15:45.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T03:15:45.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T03:15:45.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-87967-0 2024-09-15T03:15:45.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:15:45.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-0 2024-09-15T03:15:45.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-0 2024-09-15T03:15:45.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:15:45.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:15:45.819 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:15:45.819 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:15:45.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:15:45.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T03:15:45.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T03:15:45.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T03:15:45.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T03:15:45.820 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T03:15:45.821 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T03:15:45.822 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:15:45.822 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:15:45.822 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:15:45.822 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T03:15:45.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T03:15:45.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:15:45.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.3.asok 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T03:15:45.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-87967-0 0 2024-09-15T03:15:45.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87967-0 2024-09-15T03:15:45.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:15:45.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:15:45.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87967-0 2024-09-15T03:15:45.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:15:45.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:15:45.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87967-0 td/test-erasure-eio/COPY 2024-09-15T03:15:46.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:15:46.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:15:46.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-15T03:15:46.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-87967-0 2024-09-15T03:15:46.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:15:46.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-0 2024-09-15T03:15:46.062 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-0 2024-09-15T03:15:46.062 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:15:46.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:15:46.456 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:15:46.456 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:15:46.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:15:46.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-15T03:15:46.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-15T03:15:46.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-15T03:15:46.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T03:15:46.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:15:46.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:15:46.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:15:46.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:15:46.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:15:46.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:15:46.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-15T03:15:47.732 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2024-09-15T03:15:47.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-87967-0 2024-09-15T03:15:47.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-15T03:15:47.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:15:47.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-0 2024-09-15T03:15:47.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-0 2024-09-15T03:15:47.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:15:48.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:15:48.325 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:15:48.325 INFO:tasks.workunit.client.0.smithi130.stderr:0' 2024-09-15T03:15:48.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T03:15:48.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-15T03:15:49.152 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2024-09-15T03:15:49.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-15T03:15:49.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:15:49.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:15:49.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:15:49.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:15:49.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:15:49.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:15:49.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:15:49.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:15:49.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:15:49.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:15:49.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:15:49.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:15:49.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:15:49.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:15:49.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:15:49.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:15:49.168 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:15:49.168 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:15:49.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:15:49.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:15:49.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:15:49.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:15:49.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:15:49.170 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:15:49.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:15:49.171 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T03:15:49.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:15:49.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:15:49.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:15:49.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:15:49.189 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:49.188+0000 7f000871e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:49.286 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:49.284+0000 7f000871e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:49.286+0000 7f000871e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:15:49.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:15:49.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:15:49.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:15:49.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:15:49.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:15:49.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:49.565 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:15:49.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:15:49.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:49.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:49.804 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:49.803+0000 7f000871e440 -1 Falling back to public interface 2024-09-15T03:15:50.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:51.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:51.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:51.145 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:15:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:15:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:51.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:51.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:52.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:52.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:52.561 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:15:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:15:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:52.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:52.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:15:52.985 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:15:52.983+0000 7f000871e440 -1 osd.2 37 log_to_monitors true 2024-09-15T03:15:53.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:15:53.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:15:53.918 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:15:53.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:15:53.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:15:53.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:15:54.324 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 47 up_thru 19 down_at 38 last_clean_interval [19,37) [v2:127.0.0.1:6818/274827,v1:127.0.0.1:6819/274827] [v2:127.0.0.1:6820/274827,v1:127.0.0.1:6821/274827] exists,up 92b11500-291a-48bd-96e1-7e844ecb61ec 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:15:54.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:15:54.327 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:15:54.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:15:54.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:15:54.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:15:54.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:15:54.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:15:54.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:15:55.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:15:55.022 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:15:55.022 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:15:55.022 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:15:55.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:15:55.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:55.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:15:55.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803808 2024-09-15T03:15:55.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803808 2024-09-15T03:15:55.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808' 2024-09-15T03:15:55.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:55.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:15:55.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574871 2024-09-15T03:15:55.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574871 2024-09-15T03:15:55.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574871' 2024-09-15T03:15:55.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:55.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:15:55.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=201863462915 2024-09-15T03:15:55.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 201863462915 2024-09-15T03:15:55.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574871 2-201863462915' 2024-09-15T03:15:55.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:15:55.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:15:55.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084299 2024-09-15T03:15:55.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084299 2024-09-15T03:15:55.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574871 2-201863462915 3-120259084299' 2024-09-15T03:15:55.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:55.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803808 2024-09-15T03:15:55.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:55.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:15:55.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803808 2024-09-15T03:15:55.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:55.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803808 2024-09-15T03:15:55.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-15T03:15:55.588 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803808 2024-09-15T03:15:55.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:15:55.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803806 -lt 25769803808 2024-09-15T03:15:55.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:15:56.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:15:56.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:15:57.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-15T03:15:57.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:57.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574871 2024-09-15T03:15:57.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:57.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:15:57.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574871 2024-09-15T03:15:57.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:57.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574871 2024-09-15T03:15:57.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-09-15T03:15:57.294 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574871 2024-09-15T03:15:57.294 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:15:57.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574871 -lt 55834574871 2024-09-15T03:15:57.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:57.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-201863462915 2024-09-15T03:15:57.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:57.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:15:57.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-201863462915 2024-09-15T03:15:57.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:57.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=201863462915 2024-09-15T03:15:57.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-09-15T03:15:57.683 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 201863462915 2024-09-15T03:15:57.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:15:58.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 201863462915 -lt 201863462915 2024-09-15T03:15:58.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:15:58.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084299 2024-09-15T03:15:58.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:15:58.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:15:58.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084299 2024-09-15T03:15:58.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:15:58.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084299 2024-09-15T03:15:58.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084299' 2024-09-15T03:15:58.035 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 120259084299 2024-09-15T03:15:58.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:15:58.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084300 -lt 120259084299 2024-09-15T03:15:58.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:15:58.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:15:58.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:15:58.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:15:58.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:15:58.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:15:58.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:15:58.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:15:58.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:15:58.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:15:58.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:15:59.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:15:59.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:15:59.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:15:59.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:15:59.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:15:59.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:15:59.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:15:59.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-15T03:15:59.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:15:59.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T03:16:00.145 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T03:16:00.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T03:16:00.816 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T03:16:00.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T03:16:00.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:16:00.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:16:00.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:16:00.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:16:00.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:16:00.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:16:00.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:16:00.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:16:00.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:16:00.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:16:00.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:16:00.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:16:00.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:16:00.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:16:00.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:16:00.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:00.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:16:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:16:00.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:16:00.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:16:00.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:16:00.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:16:00.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:00.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:00.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:16:00.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:16:00.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:16:00.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-15T03:16:00.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:16:00.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:16:00.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:16:00.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:16:00.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:16:00.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:16:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:16:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:16:00.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:16:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:16:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:16:00.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:16:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:16:00.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:16:00.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:16:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:16:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:16:00.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:16:00.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:16:00.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-15T03:16:00.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:16:00.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:00.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:00.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.87967 2024-09-15T03:16:00.977 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:16:00.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:16:00.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-15T03:16:00.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-15T03:16:01.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:16:01.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:01.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:01.306 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:01.306 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:01.306 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:01.306 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:16:01.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:16:01.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:16:01.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:16:01.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:01.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:01.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:16:01.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:16:01.530 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:01.530 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:01.531 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:01.531 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:01.531 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:01.531 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:01.531 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:16:01.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:01.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get fsid 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:01.606 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:01.607 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:01.607 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:01.607 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:16:01.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:16:01.607 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.87967/ceph-mon.a.asok config get mon_host 2024-09-15T03:16:01.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-15T03:16:01.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-15T03:16:01.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:16:01.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:16:01.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:16:01.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-15T03:16:01.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:16:02.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:16:02.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:02.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:02.529 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:02.529 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:02.529 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:02.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:16:02.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:16:02.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:16:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-15T03:16:02.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T03:16:02.840 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'rbd' already exists 2024-09-15T03:16:02.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:16:03.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-15T03:16:03.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:16:03.853 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:16:03.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:16:03.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:03.853 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:03.854 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-mon.a.asok 2024-09-15T03:16:03.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-15T03:16:03.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-mon.a.asok log flush 2024-09-15T03:16:03.920 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:16:03.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-15T03:16:03.929 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:16:01.330+0000 7fc8e52cc900 0 load: jerasure load: lrc 2024-09-15T03:16:03.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2024-09-15T03:16:03.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2024-09-15T03:16:03.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2024-09-15T03:16:03.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:347: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2024-09-15T03:16:03.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-15T03:16:03.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-15T03:16:03.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-15T03:16:03.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-15T03:16:03.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:16:03.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-15T03:16:03.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:16:03.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:16:03.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:16:03.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:16:03.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:03.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-15T03:16:03.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:16:03.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0afec48-4cbf-4bb6-a5a1-cb1c3f3de8ea 2024-09-15T03:16:03.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c0afec48-4cbf-4bb6-a5a1-cb1c3f3de8ea' 2024-09-15T03:16:03.939 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 c0afec48-4cbf-4bb6-a5a1-cb1c3f3de8ea 2024-09-15T03:16:03.939 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:16:03.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzUeZml8aaOBAAqydbKLPuSiiSsnOUKzSwng== 2024-09-15T03:16:03.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzUeZml8aaOBAAqydbKLPuSiiSsnOUKzSwng=="}' 2024-09-15T03:16:03.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0afec48-4cbf-4bb6-a5a1-cb1c3f3de8ea -i td/test-erasure-eio/0/new.json 2024-09-15T03:16:04.236 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:16:04.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-15T03:16:04.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzUeZml8aaOBAAqydbKLPuSiiSsnOUKzSwng== --osd-uuid c0afec48-4cbf-4bb6-a5a1-cb1c3f3de8ea 2024-09-15T03:16:04.271 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:04.270+0000 7f430151f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:04.298 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:04.297+0000 7f430151f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:04.299 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:04.298+0000 7f430151f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:04.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:04.299+0000 7f430151f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:16:04.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:04.299+0000 7f430151f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:16:04.300 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:04.299+0000 7f430151f440 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-15T03:16:04.338 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:04.337+0000 7f430151f440 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-15T03:16:10.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-15T03:16:10.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:16:10.353 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-15T03:16:10.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:16:10.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:16:10.831 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-15T03:16:10.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:16:10.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:16:10.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:16:10.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:16:10.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:16:10.853 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:10.852+0000 7f5762624440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:10.855 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:10.854+0000 7f5762624440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:10.856 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:10.855+0000 7f5762624440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:11.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:16:11.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:16:11.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:16:11.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:16:11.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:16:11.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:11.180 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:16:11.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:16:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:11.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:11.376 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:11.375+0000 7f5762624440 -1 Falling back to public interface 2024-09-15T03:16:11.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:12.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:12.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:12.544 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:16:12.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:16:12.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:12.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:12.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:13.848 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:13.847+0000 7f5762624440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:16:13.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:13.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:13.907 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:16:13.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:16:13.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:13.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:14.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:15.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:15.425 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:16:15.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:16:15.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:15.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:15.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:16.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:16.805 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:16:16.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:16.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:16:16.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:16.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:17.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:18.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:18.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:18.189 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:16:18.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:16:18.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:18.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:18.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:19.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:19.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:19.666 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:16:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:16:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:19.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:20.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:21.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:21.065 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:16:21.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:21.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:16:21.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:21.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:21.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:22.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:22.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:22.546 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:16:22.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:16:22.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:22.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:22.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:23.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:23.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:23.916 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:16:23.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:16:23.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:23.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:24.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:25.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:25.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:25.295 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:16:25.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:16:25.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:25.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:26.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:27.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:27.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:27.079 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:16:27.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:16:27.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:27.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:27.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:28.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:28.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:28.569 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:16:28.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:16:28.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:28.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:29.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:30.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:30.298 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:16:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:16:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:30.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:31.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:32.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:32.161 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:16:32.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:32.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:16:32.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:32.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:32.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:33.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:33.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:33.604 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:16:33.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:16:33.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:33.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:34.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:35.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:16:35.720 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:16:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:35.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:36.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:37.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:37.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:37.131 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:16:37.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:16:37.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:37.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:16:37.535 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/276957,v1:127.0.0.1:6803/276957] [v2:127.0.0.1:6804/276957,v1:127.0.0.1:6805/276957] exists,up c0afec48-4cbf-4bb6-a5a1-cb1c3f3de8ea 2024-09-15T03:16:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:16:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:16:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:16:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:16:37.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:16:37.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:16:37.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f46fa9bf-092d-432e-bab0-9dde5999e31f 2024-09-15T03:16:37.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f46fa9bf-092d-432e-bab0-9dde5999e31f' 2024-09-15T03:16:37.543 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 f46fa9bf-092d-432e-bab0-9dde5999e31f 2024-09-15T03:16:37.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:16:37.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVUeZm//7xIBAA2AC/dEvdxFXkx001nqdDiw== 2024-09-15T03:16:37.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVUeZm//7xIBAA2AC/dEvdxFXkx001nqdDiw=="}' 2024-09-15T03:16:37.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f46fa9bf-092d-432e-bab0-9dde5999e31f -i td/test-erasure-eio/1/new.json 2024-09-15T03:16:38.029 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:16:38.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-15T03:16:38.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVUeZm//7xIBAA2AC/dEvdxFXkx001nqdDiw== --osd-uuid f46fa9bf-092d-432e-bab0-9dde5999e31f 2024-09-15T03:16:38.065 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:38.064+0000 7fbcf212f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:38.067 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:38.065+0000 7fbcf212f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:38.068 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:38.066+0000 7fbcf212f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:38.068 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:38.066+0000 7fbcf212f440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:16:38.069 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:38.066+0000 7fbcf212f440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:16:38.069 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:38.066+0000 7fbcf212f440 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-15T03:16:38.141 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:38.139+0000 7fbcf212f440 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-15T03:16:44.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-15T03:16:44.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:16:44.051 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-15T03:16:44.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:16:44.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:16:44.570 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-15T03:16:44.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:16:44.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:16:44.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:16:44.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:16:44.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:16:44.593 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:44.592+0000 7f4dd5a28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:44.595 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:44.594+0000 7f4dd5a28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:44.596 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:44.595+0000 7f4dd5a28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:16:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:16:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:16:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:16:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:16:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:16:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:44.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:16:44.918 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:16:44.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:44.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:45.115 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:45.114+0000 7f4dd5a28440 -1 Falling back to public interface 2024-09-15T03:16:45.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:46.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:46.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:46.274 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:16:46.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:16:46.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:46.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:46.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:47.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:47.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:47.625 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:16:47.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:16:47.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:47.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:47.629 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:16:47.628+0000 7f4dd5a28440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:16:48.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:49.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:49.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:49.081 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:16:49.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:16:49.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:49.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:49.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:50.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:50.502 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:16:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:16:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:50.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:50.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:51.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:51.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:51.972 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:16:51.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:16:51.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:51.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:52.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:53.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:53.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:53.457 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:16:53.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:16:53.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:53.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:53.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:54.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:54.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:54.853 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:16:54.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:16:54.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:54.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:55.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:56.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:56.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:56.216 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:16:56.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:16:56.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:56.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:56.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:57.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:57.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:57.861 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:16:57.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:16:57.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:57.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:58.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:16:59.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:16:59.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:16:59.221 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:16:59.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:16:59.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:16:59.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:16:59.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:00.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:00.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:00.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:17:00.803 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:17:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:00.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:01.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:02.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:02.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:02.531 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:17:02.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:17:02.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:02.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:03.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:04.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:04.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:04.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:17:04.703 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:17:04.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:04.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:05.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:06.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:06.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:06.182 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:17:06.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:17:06.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:06.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:06.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:07.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:07.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:07.637 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:17:07.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:17:07.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:07.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:08.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:09.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:09.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:09.486 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:17:09.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:17:09.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:09.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:10.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:11.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:11.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:11.370 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:17:11.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:17:11.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:11.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:17:11.724 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/278711,v1:127.0.0.1:6811/278711] [v2:127.0.0.1:6812/278711,v1:127.0.0.1:6813/278711] exists,up f46fa9bf-092d-432e-bab0-9dde5999e31f 2024-09-15T03:17:11.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:17:11.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:17:11.729 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:17:11.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:17:11.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:17:11.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:17:11.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:17:11.731 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:17:11.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98ff4a4a-2640-4d53-87c9-0ceb1d289f08 2024-09-15T03:17:11.731 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 98ff4a4a-2640-4d53-87c9-0ceb1d289f08 2024-09-15T03:17:11.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 98ff4a4a-2640-4d53-87c9-0ceb1d289f08' 2024-09-15T03:17:11.732 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:17:11.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3UeZmRoAsLBAAYPvku8LGPF3TZnMfjE98OQ== 2024-09-15T03:17:11.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3UeZmRoAsLBAAYPvku8LGPF3TZnMfjE98OQ=="}' 2024-09-15T03:17:11.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98ff4a4a-2640-4d53-87c9-0ceb1d289f08 -i td/test-erasure-eio/2/new.json 2024-09-15T03:17:12.342 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:17:12.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-15T03:17:12.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3UeZmRoAsLBAAYPvku8LGPF3TZnMfjE98OQ== --osd-uuid 98ff4a4a-2640-4d53-87c9-0ceb1d289f08 2024-09-15T03:17:12.377 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:12.376+0000 7f9e04411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:12.419 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:12.418+0000 7f9e04411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:12.420 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:12.419+0000 7f9e04411440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:12.420 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:12.419+0000 7f9e04411440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:17:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:12.419+0000 7f9e04411440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:17:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:12.419+0000 7f9e04411440 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-15T03:17:12.459 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:12.458+0000 7f9e04411440 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-15T03:17:18.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-15T03:17:18.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:17:18.313 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-15T03:17:18.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:17:18.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:17:18.802 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:17:18.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:17:18.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:17:18.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:17:18.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:17:18.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:17:18.825 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:18.824+0000 7f973b009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:18.856 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:18.855+0000 7f973b009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:18.857 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:18.856+0000 7f973b009440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:19.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:17:19.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:17:19.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:17:19.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:17:19.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:17:19.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:19.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:17:19.185 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:17:19.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:19.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:19.386 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:19.385+0000 7f973b009440 -1 Falling back to public interface 2024-09-15T03:17:19.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:20.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:20.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:20.537 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:17:20.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:17:20.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:20.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:20.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:21.858 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:21.857+0000 7f973b009440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:17:21.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:21.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:21.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:17:21.925 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:17:21.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:21.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:22.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:23.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:23.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:23.511 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:17:23.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:17:23.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:23.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:23.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:24.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:24.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:24.957 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:17:24.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:17:24.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:24.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:25.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:26.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:26.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:26.389 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:17:26.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:17:26.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:26.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:26.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:27.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:27.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:27.795 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:17:27.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:17:27.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:27.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:28.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:29.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:29.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:29.209 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:17:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:17:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:29.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:29.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:30.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:30.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:30.641 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:17:30.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:17:30.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:30.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:31.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:32.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:32.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:32.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:17:32.171 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:17:32.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:32.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:32.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:33.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:33.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:33.646 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:17:33.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:17:33.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:33.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:34.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:35.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:35.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:17:35.016 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:17:35.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:35.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:36.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:37.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:37.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:37.139 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:17:37.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:17:37.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:37.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:37.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:38.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:38.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:38.925 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:17:38.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:17:38.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:38.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:39.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:40.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:40.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:40.579 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:17:40.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:17:40.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:40.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:41.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:42.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:42.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:42.580 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:17:42.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:17:42.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:42.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:43.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:44.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:44.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:44.340 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:17:44.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:17:44.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:44.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:45.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:46.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:46.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:46.275 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:17:46.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:17:46.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:46.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:17:46.835 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/280465,v1:127.0.0.1:6819/280465] [v2:127.0.0.1:6820/280465,v1:127.0.0.1:6821/280465] exists,up 98ff4a4a-2640-4d53-87c9-0ceb1d289f08 2024-09-15T03:17:46.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:17:46.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:17:46.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:17:46.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=338a2ea7-4a7a-41c3-a845-636e4ab58818 2024-09-15T03:17:46.845 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 338a2ea7-4a7a-41c3-a845-636e4ab58818 2024-09-15T03:17:46.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 338a2ea7-4a7a-41c3-a845-636e4ab58818' 2024-09-15T03:17:46.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:17:46.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaUeZmMvfbMhAAnbl0A2C4HN4oEnLcUVuVFw== 2024-09-15T03:17:46.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaUeZmMvfbMhAAnbl0A2C4HN4oEnLcUVuVFw=="}' 2024-09-15T03:17:46.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 338a2ea7-4a7a-41c3-a845-636e4ab58818 -i td/test-erasure-eio/3/new.json 2024-09-15T03:17:47.402 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:17:47.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-15T03:17:47.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaUeZmMvfbMhAAnbl0A2C4HN4oEnLcUVuVFw== --osd-uuid 338a2ea7-4a7a-41c3-a845-636e4ab58818 2024-09-15T03:17:47.438 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:47.437+0000 7f2856c21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:47.440 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:47.439+0000 7f2856c21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:47.441 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:47.440+0000 7f2856c21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:47.442 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:47.441+0000 7f2856c21440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:17:47.442 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:47.441+0000 7f2856c21440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:17:47.442 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:47.441+0000 7f2856c21440 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-15T03:17:47.518 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:47.517+0000 7f2856c21440 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-15T03:17:53.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-15T03:17:53.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:17:53.930 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-15T03:17:53.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T03:17:53.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:17:54.413 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-15T03:17:54.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T03:17:54.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:17:54.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:17:54.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:17:54.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:17:54.436 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:54.434+0000 7fc8b9a6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:54.438 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:54.436+0000 7fc8b9a6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:54.439 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:54.437+0000 7fc8b9a6e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:17:54.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T03:17:54.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:17:54.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T03:17:54.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:17:54.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:17:54.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:54.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:17:54.804 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:17:54.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:54.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:17:54.963 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:54.962+0000 7fc8b9a6e440 -1 Falling back to public interface 2024-09-15T03:17:55.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:56.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:56.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:56.158 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:17:56.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:17:56.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:56.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:17:56.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:57.430 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:17:57.429+0000 7fc8b9a6e440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:17:57.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:57.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:57.519 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:17:57.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:17:57.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:57.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:17:58.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:17:59.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:17:59.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:17:59.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:17:59.022 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:17:59.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:17:59.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:17:59.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:00.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:00.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:00.390 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-15T03:18:00.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:18:00.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:00.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:00.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:01.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:01.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:01.956 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-15T03:18:01.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:18:01.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:01.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:02.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:03.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:03.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:03.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:18:03.436 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-15T03:18:03.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:03.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:03.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:04.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:04.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:04.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:18:04.850 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-15T03:18:04.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:04.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:05.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:06.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:06.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:06.406 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-15T03:18:06.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:18:06.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:06.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:06.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:07.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:07.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:07.931 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-15T03:18:07.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:18:07.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:07.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:08.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:09.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:09.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:09.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:18:09.452 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-15T03:18:09.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:09.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:09.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:10.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:10.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:10.849 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-15T03:18:10.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:18:10.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:10.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:11.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:12.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:12.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:12.517 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-15T03:18:12.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:18:12.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:12.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:13.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:14.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:14.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:14.241 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-15T03:18:14.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:18:14.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:14.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:14.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:15.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:15.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:15.823 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-15T03:18:15.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:18:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:15.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:16.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:17.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:17.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:17.605 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-15T03:18:17.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:18:17.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:17.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:18.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:19.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:19.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:19.400 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-15T03:18:19.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:18:19.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:19.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:20.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:21.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:21.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:21.563 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-15T03:18:21.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:18:21.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:21.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:22.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:23.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:23.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:23.133 INFO:tasks.workunit.client.0.smithi130.stdout:18 2024-09-15T03:18:23.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:18:23.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:23.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T03:18:23.590 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/282219,v1:127.0.0.1:6827/282219] [v2:127.0.0.1:6828/282219,v1:127.0.0.1:6829/282219] exists,up 338a2ea7-4a7a-41c3-a845-636e4ab58818 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:18:23.592 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:18:23.593 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.0.asok 2024-09-15T03:18:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-15T03:18:23.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.0.asok log flush 2024-09-15T03:18:23.668 INFO:tasks.workunit.client.0.smithi130.stdout:{} 2024-09-15T03:18:23.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-15T03:18:23.677 INFO:tasks.workunit.client.0.smithi130.stdout:2024-09-15T03:16:11.379+0000 7f5762624440 0 load: jerasure load: lrc 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:349: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-15T03:18:23.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-15T03:18:23.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-15T03:18:24.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-15T03:18:24.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-15T03:18:25.416 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' already exists 2024-09-15T03:18:25.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:18:26.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-15T03:18:26.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:18:26.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:18:26.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:18:26.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:18:26.429 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:18:26.430 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:18:26.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:18:26.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:18:26.430 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:18:26.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:18:26.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:18:26.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:18:26.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:18:26.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:18:26.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:18:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:18:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:18:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:18:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:18:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:18:26.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:26.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:18:27.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803807 2024-09-15T03:18:27.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803807 2024-09-15T03:18:27.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807' 2024-09-15T03:18:27.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:27.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:18:27.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574869 2024-09-15T03:18:27.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574869 2024-09-15T03:18:27.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807 1-55834574869' 2024-09-15T03:18:27.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:27.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:18:27.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378638 2024-09-15T03:18:27.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378638 2024-09-15T03:18:27.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807 1-55834574869 2-81604378638' 2024-09-15T03:18:27.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:27.283 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:18:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051589 2024-09-15T03:18:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051589 2024-09-15T03:18:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807 1-55834574869 2-81604378638 3-124554051589' 2024-09-15T03:18:27.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:27.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803807 2024-09-15T03:18:27.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:27.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:18:27.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803807 2024-09-15T03:18:27.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:27.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803807 2024-09-15T03:18:27.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-09-15T03:18:27.393 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803807 2024-09-15T03:18:27.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:18:27.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803805 -lt 25769803807 2024-09-15T03:18:27.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:18:28.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:18:28.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:18:29.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803807 2024-09-15T03:18:29.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:29.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574869 2024-09-15T03:18:29.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:29.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:18:29.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574869 2024-09-15T03:18:29.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574869 2024-09-15T03:18:29.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574869' 2024-09-15T03:18:29.131 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574869 2024-09-15T03:18:29.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:18:29.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574869 -lt 55834574869 2024-09-15T03:18:29.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:29.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378638 2024-09-15T03:18:29.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:29.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:18:29.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378638 2024-09-15T03:18:29.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:29.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378638 2024-09-15T03:18:29.494 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 81604378638 2024-09-15T03:18:29.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378638' 2024-09-15T03:18:29.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:18:29.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378638 -lt 81604378638 2024-09-15T03:18:29.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:29.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051589 2024-09-15T03:18:29.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:29.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:18:29.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051589 2024-09-15T03:18:29.847 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:29.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051589 2024-09-15T03:18:29.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051589' 2024-09-15T03:18:29.849 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 124554051589 2024-09-15T03:18:29.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:18:30.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051590 -lt 124554051589 2024-09-15T03:18:30.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:18:30.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:30.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:18:30.668 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:18:31.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:18:31.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:18:31.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:31.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:31.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:352: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-15T03:18:31.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-87967-1 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-87967-1 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-87967-1 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-15T03:18:31.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T03:18:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-15T03:18:31.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-87967-1 td/test-erasure-eio/ORIGINAL 2024-09-15T03:18:31.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-87967-1 td/test-erasure-eio 1 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-87967-1 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T03:18:31.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T03:18:31.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-87967-1 2024-09-15T03:18:31.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:18:31.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-1 2024-09-15T03:18:31.648 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-1 2024-09-15T03:18:31.648 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '2') 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-15T03:18:32.050 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:18:32.052 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:18:32.053 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:18:32.053 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:32.053 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:18:32.053 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T03:18:32.053 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.87967/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T03:18:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:18:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T03:18:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.87967/ceph-osd.1.asok 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T03:18:32.134 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.87967/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-87967-1 1 2024-09-15T03:18:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-87967-1 2024-09-15T03:18:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-15T03:18:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-15T03:18:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-87967-1 2024-09-15T03:18:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-15T03:18:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-15T03:18:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-87967-1 td/test-erasure-eio/COPY 2024-09-15T03:18:32.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-15T03:18:32.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-15T03:18:32.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-15T03:18:32.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-87967-1 2024-09-15T03:18:32.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:18:32.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-1 2024-09-15T03:18:32.268 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-1 2024-09-15T03:18:32.268 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:18:32.623 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:18:32.624 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:18:32.624 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-15T03:18:32.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-15T03:18:32.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-09-15T03:18:32.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-15T03:18:32.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-15T03:18:32.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-15T03:18:32.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:18:32.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:18:32.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:18:32.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:18:32.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:18:32.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:18:32.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-15T03:18:33.920 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already out. 2024-09-15T03:18:33.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-87967-1 2024-09-15T03:18:33.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-15T03:18:33.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-15T03:18:33.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-87967-1 2024-09-15T03:18:33.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-87967-1 2024-09-15T03:18:33.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:18:34.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T03:18:34.434 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:18:34.434 INFO:tasks.workunit.client.0.smithi130.stderr:0' 2024-09-15T03:18:34.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T03:18:34.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-15T03:18:35.375 INFO:tasks.workunit.client.0.smithi130.stderr:osd.2 is already in. 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-15T03:18:35.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:18:35.390 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-15T03:18:35.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:18:35.392 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-15T03:18:35.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32f719bc-02d9-4ef8-b6b6-3b20838ee507 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.87967/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:18:35.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-15T03:18:35.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:18:35.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:18:35.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:18:35.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:18:35.411 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:18:35.410+0000 7fdb88f66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:35.413 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:18:35.412+0000 7fdb88f66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:35.414 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:18:35.413+0000 7fdb88f66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:18:35.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:18:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:18:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:18:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:18:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:18:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:35.907 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-15T03:18:35.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:18:35.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:35.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:18:35.949 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:18:35.948+0000 7fdb88f66440 -1 Falling back to public interface 2024-09-15T03:18:36.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:37.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:37.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:37.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:18:37.489 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-15T03:18:37.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:37.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:18:37.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:38.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:38.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:38.849 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-15T03:18:38.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:18:38.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:38.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:18:39.142 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-15T03:18:39.141+0000 7fdb88f66440 -1 osd.2 38 log_to_monitors true 2024-09-15T03:18:39.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:18:40.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:18:40.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:18:40.222 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-15T03:18:40.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:18:40.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:18:40.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:18:40.787 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 49 up_thru 19 down_at 39 last_clean_interval [19,38) [v2:127.0.0.1:6818/283796,v1:127.0.0.1:6819/283796] [v2:127.0.0.1:6820/283796,v1:127.0.0.1:6821/283796] exists,up 98ff4a4a-2640-4d53-87c9-0ceb1d289f08 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:18:40.789 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:18:40.790 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:18:40.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:18:40.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:18:40.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:18:40.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:18:40.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:18:40.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:18:40.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:18:40.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:18:40.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:18:41.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:18:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-15T03:18:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-15T03:18:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:3' 2024-09-15T03:18:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:18:41.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:41.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:18:41.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803812 2024-09-15T03:18:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803812 2024-09-15T03:18:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812' 2024-09-15T03:18:41.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:41.608 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:18:41.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574874 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574874 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-55834574874' 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:41.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:18:41.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397507 2024-09-15T03:18:41.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397507 2024-09-15T03:18:41.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-55834574874 2-210453397507' 2024-09-15T03:18:41.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:18:41.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T03:18:42.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051594 2024-09-15T03:18:42.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051594 2024-09-15T03:18:42.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-55834574874 2-210453397507 3-124554051594' 2024-09-15T03:18:42.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:42.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803812 2024-09-15T03:18:42.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:42.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:18:42.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803812 2024-09-15T03:18:42.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:42.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803812 2024-09-15T03:18:42.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-15T03:18:42.034 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 25769803812 2024-09-15T03:18:42.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:18:42.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-15T03:18:42.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:42.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574874 2024-09-15T03:18:42.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:42.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:18:42.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574874 2024-09-15T03:18:42.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:42.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574874 2024-09-15T03:18:42.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-09-15T03:18:42.392 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 55834574874 2024-09-15T03:18:42.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:18:42.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574874 -lt 55834574874 2024-09-15T03:18:42.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:42.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-210453397507 2024-09-15T03:18:42.744 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:42.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:18:42.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-210453397507 2024-09-15T03:18:42.746 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:42.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397507 2024-09-15T03:18:42.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-09-15T03:18:42.747 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 210453397507 2024-09-15T03:18:42.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:18:43.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-15T03:18:43.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:18:43.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051594 2024-09-15T03:18:43.112 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:18:43.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T03:18:43.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051594 2024-09-15T03:18:43.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:18:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051594 2024-09-15T03:18:43.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051594' 2024-09-15T03:18:43.115 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 124554051594 2024-09-15T03:18:43.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T03:18:43.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051594 -lt 124554051594 2024-09-15T03:18:43.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:18:43.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:43.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:43.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-15T03:18:43.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:18:43.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:18:43.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:18:43.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:18:43.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:18:43.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:18:43.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:18:44.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T03:18:44.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:18:44.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:18:44.278 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:18:44.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-15T03:18:44.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:18:44.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:18:44.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:354: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-15T03:18:44.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-15T03:18:44.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-15T03:18:45.215 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'pool-jerasure' does not exist 2024-09-15T03:18:45.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-15T03:18:45.875 INFO:tasks.workunit.client.0.smithi130.stderr:erasure-code-profile myprofile does not exist 2024-09-15T03:18:45.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-15T03:18:45.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:18:45.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:18:45.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:18:45.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:18:45.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:18:45.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:18:45.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:18:45.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:18:45.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:18:45.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:18:45.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:18:45.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:18:45.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:18:45.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:18:45.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:18:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:18:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:18:45.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:18:45.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:18:46.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:18:46.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:18:46.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:18:46.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:18:46.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:46.022 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:18:46.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:18:46.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:18:46.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:18:46.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-erasure-eio 0 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:18:46.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:18:46.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:18:46.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:18:46.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:18:46.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:18:46.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:18:46.028 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:18:46.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:18:46.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:18:46.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:18:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:18:46.030 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:18:46.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:18:46.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T03:18:46.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-15T03:18:46.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:18:46.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:18:46.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.87967 2024-09-15T03:18:46.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.87967 2024-09-15T03:18:46.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:18:46.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:18:46.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T03:18:46.035 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:18:46.035 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:18:46.062 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-09-15T03:18:46.062 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T03:18:46.366 DEBUG:teuthology.parallel:result is None 2024-09-15T03:18:46.366 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:18:46.390 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:18:46.391 DEBUG:teuthology.orchestra.run.smithi130:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:18:46.461 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:18:46.461 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T03:18:46.470 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T03:18:46.471 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:18:46.552 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T03:18:46.561 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T03:18:46.561 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:18:46.580 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:18:46.584 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:18:46.584 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-09-15T03:18:46.585 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv01.front.sepia.ceph.com 3 6 377 55 +232us[ +232us] +/- 48ms 2024-09-15T03:18:46.585 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 60 -239us[ -240us] +/- 76ms 2024-09-15T03:18:46.585 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -98us[ -98us] +/- 84ms 2024-09-15T03:18:46.585 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:18:46.585 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T03:18:46.593 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T03:18:46.594 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T03:18:46.604 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T03:18:46.684 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:18:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:18:46.711 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T03:18:46.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T03:18:46.729 INFO:teuthology.task.internal:Duration was 4813.699476 seconds 2024-09-15T03:18:46.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T03:18:46.738 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T03:18:46.738 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:18:46.789 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:18:47.162 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T03:18:47.162 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-09-15T03:18:47.162 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:18:47.228 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T03:18:47.228 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:18:47.295 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T03:18:47.295 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:18:47.432 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T03:18:47.442 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T03:18:47.442 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:18:47.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T03:18:47.476 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:18:47.530 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-09-15T03:18:47.543 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:18:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:18:47.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T03:18:47.607 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T03:18:47.609 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905101/remote/smithi130 2024-09-15T03:18:47.610 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:18:47.685 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T03:18:47.685 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:18:47.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T03:18:47.745 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T03:18:47.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T03:18:47.798 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T03:18:47.798 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:18:47.813 INFO:teuthology.orchestra.run.smithi130.stdout: 265861 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:18 /home/ubuntu/cephtest 2024-09-15T03:18:47.814 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T03:18:47.825 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T03:18:47.847 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} duration: 4813.699476480484 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T03:18:47.847 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:18:47.940 INFO:teuthology.run:pass